Has anyone uses nvmadm to updated an intel drive?

I'm sure we must have at Joyent

I couldn't get a repeatable load into a slot

So I had to resort to bouncing into the updater via virtual media. I've got a server where we can test if need be.

Smithx10: Do you have whatever error output it generated?

LeftWing: It was during the commit, I'll try to reproduce again tomorrow.

get some more infor

Ok cool

Actually, Ill try to do it on a node real quick

welp, this one got further and then made my disk 0.0 gb "_"

<3

I bounced and loaded up the firmware tool from intel, it showed only 2 of the drives needed flashing. Guessing having that disk in a pool was not good.... but activate suggests that you need to power cycle to have that firmware kick in

pxe booting now, hopefully all is well

good news, the drive and pool appear to be healthy... activate did some :P :P Are we supposed to only use nvmeadm under certain conditions?

An excellent question, to which I do not presently have the answer

It does seem, from the manual, like it's just a configuration thing that won't take effect until the next controller rest

reset

FYI, I have decided Twitter is probably not a fad!

twitter.com/illumosorg -- I demand followers! :D

LeftWing following as ordered

LeftWing: already following ;)

rmustacc: I uploaded the crash dump and would like to share the vmdump.3 with just a few people (it surely contains private data), so when you are here give me a ping please and I'll share the url with you

Agnar_: How big is it

6.1GB

That's not too bad

uncompressed it's 18.1G

I should really make a dump upload server thing

Also, I miss Manta.

rip

jk it still runs fine

there's always minimanta

minimanta?

my shitty mostly-manta-API-compatible thing which stores the data as plain files on disk and puts the metadata in the resource fork

basically if you want to be able to use the manta tools and msign etc against just a plain directory on a single box somewhere, it's that

should we donate capacity?

our capacity is in .au though

slow for everyone except us haha

anything free is worth what you pay for it

and then we'll find out these vmdumps are really miners

well if LeftWing (IRC) wants a manta on the moon to store shiz in for a dump upload service he knows where to find us

yep

You guys should put up some project SPARC boxes in your outback data cupboard

I guess 200ms away is not long enough for the moon, but it's further than geostationary orbit at least

SPARC boxes can't respond to an SSH packet in less than 200ms anyway

LeftWing: i do have a serious offer from a friend in alice springs to host gear

Haha think of the dust

I wonder if we would boot on one of those

I did seriously consider bringing some Cool Threads home with me

LeftWing: good news on sparc btw. I got the information that the T4s got free, the applications got migrated. Now we wait for the official "ok" to shut them down. I'll get three T4-4, 2sockets each, 256G RAM each. I'll make two out of three and I plan to bring the resulting 4 socket, 512G one into a proper datacenter

there was code for them

arekinath1: Because you didn't take them home, Samsung destroyed them

yeah well our bags were too full as it was

just ask the qantas people who had to help us with them

Ha

dlg@opiate ~$ /usr/bin/time ssh 130.102.96.36 sysctl hw.product

hw.product=SUNW,SPARC-Enterprise

0.65 real 0.01 user 0.00 sys

grumble

i blame pivy for most of that time

lol

yk5 ec signature takes about 70-80ms

you using rsa?

how dare you

debug1: kex: algorithm: curve25519-sha256

debug1: kex: host key algorithm: ecdsa-sha2-nistp256

debug1: kex: server->client cipher: chacha20-poly1305⊙oc MAC: <implicit> compression: none

even rsa2048 is only about 140ms on a yk5, and that's with all the pivy-agent overhead in ;)

using an ed25519 key on disk is about 0.45 real

ANYWAY

sparc is fine

time to run that same command is about the same this amd seattle box

not slow at all

yeah ssh to the machine next to the one I'm typing on often takes 200ms

it has to do a bunch of roundtrips and thinking and then start a pty and shell

bunch of work

The control socket advantage is real

uh huh

At least for establishment

true

LeftWing: re dump upload: sort of "supportfiles.sun.com"?

LeftWing: btw, I was about to write a kind of a generic "explorer" for illumos systems...do you see a need for that?

Just curious: who owns this account: twitter.com/Illumos_org?

nanxiao: LeftWing I think

based on his demand for followers :p

Smithx10: are those NVMe's in a mirror?

If so, you could probably zpool offline them before flashing?

Thats what I did when I had to move some disks around in a chassis, offline it, pull it, plug it, online it, wait for resilver, repeat

scrub

sjorge: @illumosorg is the new one that LeftWing mentioned, @Illumos_org (note with the incorrect capital I) - no idea

didn't even notice the _

maybe we could get the one from leftwing the verified checkmark

somehow

sjorge: sadly these are in raidz

Smithx10 I suppose technically you can offline these too but... hmmm... seems more risky than mirror to me

sjorge: I'm not sure, but... it looked like you needed to power cycle to the device to get it to use the firmware in the new slot

not sure how it is supposed to behave

got friends asking about a Solaris 10u11 system, it fails to mount a ZFS pool from an iSCSI NAS after some outage on that NAS. It was even zfs-mirrored over two volumes from that NAS, and with a separate log-device mirrored, but but when `zpool import` processes it (even in `-F -n` mode to try rolling back), the kernel panics

assertion failed: 0 == dsl_dataset_hold_obj(<args>) (0x0 == 0x32), file: .../dsl_scan.c, line: 1359

jimklimov I'd use 11.4 media

I suppose I can try to walk the pool halves with zdb to see if any is okay, but not sure over the years which commands can do effectively the userspace equivalent of import and show what is the error without a kernel panic

or maybe import without the log devices, but no idea how to check if they are the problem ")

or a new distro indeed?

do you mean booting with new live media and trying to import?

can live media be an iscsi client? :)

prepare new media with client :)

well, the point behind 11.4 is simple... you get no patches for 10 and this kind of assert smells for patch.

there is a price for running ancient software.

tsoome: aside from the bootparams that are passed to the kernel, is there any other mechanism of passing data to the kernel?

loader env is

And there are more general multiboot tags.

There's a lot of data that gets to dboot/unix, but they have to decide to do something with it.

I have used just module for loader env, and console font

Agnar_: I think I have finally replicated your postgres HEAD hang..

Agnar_: which process was it that caused the panic when you killed it? server or client

Heya, extremely noob question here, but I saw illumos.org/books/dev/workflow.html…r-best-friend-while-building-bldenv has a link to the bldenv man page, but that link appears broken

I can tell that bldenv starts a new shell, and that it helps me do incremental builds (somehow?) but is there something I can read for more detail on what it's actually doing?

sean____: yes. use usr/src/tools/scripts/bldenv

bldenv mostly just sets a _lot_ of environment variables for the shell it starts

stuff that the build will want/need

sean____: nightly -i is perhaps what you need

i would _strongly_ suggest doing a full build before attempting any sort of incremental build

jbk: Yup, I did (and booted into the new BE successfully), now trying to work through the incremental flow

there are unfortunately more rough edges around incremental builds than i think any of us would like (never enough hours in the day)... so just be prepared

(sometimes it requires a bit more knowledge of the build internals to get it to do the right thing when incrementally building, which ideally shouldn't be necessary)

but we can try to help sort through that too

heh I'm not sure I've ever worked with a system that hasn't had rough edges around incremental rebuilds in *some* cases, so I'll mentally prep myself :)

i always find it's good to set expectations.. and to make sure you know help is available :)

tsoome: i was thinking about ways we could pass a key to the kernel from the bootloader for an encrypted root pool (granted I suspect the boot files in the pool would still need to be on unencrypted datasets)

atm not just the boot files but also /platform

IMO there are 2 possible ways, use the normal module approach with specific name, as we do for env/font; or invent special structure and push it to specification.

both are doable

does the bootloader construct those modules dynamically?

some sort of structured approach is needed anyhow, but, now we can encode nvlists in loader

yes

ahh ok

in common/module.c build_font_module() and build_environment_module()

module approach is handy because we already have code in kernel to release modules, only need to add secure memory wipe too:)

i've been trying to get details on how linux or windows handles whole disk encryption of their boot disks -- as best as i can tell, there's some small unencrypted partition that it starts from

for zfs, we have 2 cases, loader is loaded from ESP (UEFI), it is not encrypted, loader can ask for pass/key and go on. And for x86 we have gptzfsboot in pool label area, which is also not encrypted.

so that part is already good

just need to add decrypt for loader zfs reader.

+ key transport and pickup mechanism for kernel.

i'm surprised there hasn't been any work on the fbsd for this.. seems like something someone would have jumped on by now :)

err fbsd side of the loader

I have it in my queue:D

in todo queue, that is.

There are not too many candidates who might pick the task:D

and most are busy preparing 13 for now

.oO and I am writing instructions for a customer, how to migrate and decommission old brocade FC switches:D

tsoome: `pkg update` doesn't take care of updating the ESP bits?

but anyhow, ability to read encrypted pools is almost in top of my list

ypankov it should do

but it depensd on how old is the system

"it" in this context means relatively up to date system

beadm activate also should.

both are using the libbe.so for that task anyhow

ah, could it be that 7357 did not update the usr/src/boot/Makefile.version? (still fighting with that change :D)

andyf: it hangs on the client and if I kill them, the system panics

ypankov hm? git diff...

i mean, if Makefile.version was not updated, libbe does not update ESP?

yes, the update is based on version number

yep, increasing that version made it update ESP, thanks

was it missing?

yep, as i said, 7357 didn't update the version (and i'm merging upto that change)

it's probably fine with all the updates that were after that

7537 :D but, you also want 13345 if there are vdevs removed from pool...

[illumos-gate] 13628 sppptun: array subscript 0 is outside array bounds -- Toomas Soome <tsoome⊙mc>

right, i can't spell

tsoome: any plans on efibootmgr-like implementation?

eventually

damn, another Fast Data Access MMU Miss. time to walk my dog:)

[illumos-gate] 13697 zfs change-key does not follow clones, data loss ensues -- Tom Caputi <tcaputi⊙dc>

[illumos-gate] 13601 ksh shell lint misleading -- Andy Fiddaman <omnios⊙ccu>

letsencrypt.org/2021/02/10/200m-certs-24hrs.html - neat! :)

Our previous generation of database servers had dual Intel Xeon E5-2650 v4 CPUs, 24 physical cores in total. They had 1TB memory with 24 3.8TB SSDs connected via SATA in a RAID 10 configuration. These worked fine for daily issuance but would not be able to handle replacing all of our certificates in a single day. We have replaced them with a new generation of database server from Dell featuring dual AMD EPYC 7542 CPUs, 64 ph

cores in total.

These machines have 2TB of faster RAM. Much faster CPUs and double the memory is great, but the really interesting thing about these machines is that the EPYC CPUs provide 128 PCIe4 lanes each. This means we could pack in 24 6.4TB NVME drives for massive I/O performance. There is no viable hardware RAID for NVME, so we’ve switched to ZFS to provide the data protection we need.

Nice :)

Dell was always a problem because of the RAID cards (unless you could get 'Nexenta spec'), but with NVMe it's much nicer

i've deployed a big'ol zfs storage with dell r740xd recently. 2xSSD, 22xNVME, freebsd13+zfs

approx 25TB raw capacirty

Does anyone have any clues on what these two lines are about? github.com/illumos/illumos-gate/blo…ts/common/exec/elf/elf_notes.c#L379

I don't understand why, if neither FSEARCH nor FEXEC are set, it basically sets loads of bits on

directory permissions?

maybe (just guessing) since it's recording the open files, the idea is distinguishing between mapped and just open for read/write?

you can open file (by name) when you can not search the directory content

No, this is something a bit weird.

It adding it seems suspect, but the prolem is that there's basically two different sets of flags in the file_t.

Look at f_flag an f_flag2.

And there's a bunch of things around how FSEARCH|FEXEC deal with that pattern.

Yes, flag2 being the 'open only' ones

Not saying I undersatnd it, but it's usually related to that.

and the bottom few bits of open() flags are mutually exclusive

Whether that should be += or &= I'm not sure.

but for some reason, when dumping the info to a core file, it ends up adding 0xffffffff if neither flag is et

but for some reason, when dumping the info to a core file, it ends up adding 0xffffffff if neither flag is set

and yes, += seemed odd too

(the same thing happens in /proc/xxx/fdinfo, but that's because I based that on this stuff

Though this is what F_GETFL does.

The += would cause things to overflow if bits aren't set. But I don't know enough of the context here.

ah right, it's in F_GETFL too

It'll require a bit of archaeology to figure out, but I suspect raf had a plan.

regarding my earlier post, here's HW-centric details (and improvement graphs) of LE current generation of 24xNVMe EPYC servers: letsencrypt.org/2021/01/21/next-gen-database-servers.html and their setup overview from getting feet wet with OpenZFS: github.com/letsencrypt/openzfs-nvme-databases

rmustacc - ok, thanks - at least I am not missing anything obvious.

it has an interesting affect on `pfiles` output - lots of flags set

*effect

andyf: Yeah. I'm not quite sure there. We'll have to dig.

It does seem suspect on pfiles.

well, same with `elfdump -n` on a core

but I suppose that information is there for `pfiles` ultimately..

We'll just need to come to an understanding of that and leave some comments for future us.

Is there a recommended gitignore for illumos-gate? after running the build, "git status" is returning hundreds of new files in varying locations with many different suffixes

sean____: I think folks tend to "git config status.showUntrackedFiles no"

Though it would certainly be good to have a .gitignore that was kept up-to-date

This is gonna sound stupid, but I would expect that to cause problems for people sending commits for review, and accidentally leaving things out - does that not happen in practice?

It would be a lie to say it has never happened haha

We try pretty hard to make sure a full clean build is done before integrating a change