-
Smithx10
Has anyone uses nvmadm to updated an intel drive?
-
LeftWing
I'm sure we must have at Joyent
-
Smithx10
I couldn't get a repeatable load into a slot
-
Smithx10
So I had to resort to bouncing into the updater via virtual media. I've got a server where we can test if need be.
-
LeftWing
Smithx10: Do you have whatever error output it generated?
-
Smithx10
LeftWing: It was during the commit, I'll try to reproduce again tomorrow.
-
Smithx10
get some more infor
-
LeftWing
Ok cool
-
Smithx10
Actually, Ill try to do it on a node real quick
-
Smithx10
-
Smithx10
welp, this one got further and then made my disk 0.0 gb "_"
-
Smithx10
<3
-
Smithx10
I bounced and loaded up the firmware tool from intel, it showed only 2 of the drives needed flashing. Guessing having that disk in a pool was not good.... but activate suggests that you need to power cycle to have that firmware kick in
-
Smithx10
pxe booting now, hopefully all is well
-
Smithx10
good news, the drive and pool appear to be healthy... activate did some :P :P Are we supposed to only use nvmeadm under certain conditions?
-
LeftWing
An excellent question, to which I do not presently have the answer
-
LeftWing
It does seem, from the manual, like it's just a configuration thing that won't take effect until the next controller rest
-
LeftWing
reset
-
LeftWing
FYI, I have decided Twitter is probably not a fad!
-
LeftWing
twitter.com/illumosorg -- I demand followers! :D
-
v_a_b
LeftWing following as ordered
-
Agnar_
LeftWing: already following ;)
-
Agnar_
rmustacc: I uploaded the crash dump and would like to share the vmdump.3 with just a few people (it surely contains private data), so when you are here give me a ping please and I'll share the url with you
-
LeftWing
Agnar_: How big is it
-
Agnar_
6.1GB
-
LeftWing
That's not too bad
-
Agnar_
uncompressed it's 18.1G
-
LeftWing
I should really make a dump upload server thing
-
LeftWing
Also, I miss Manta.
-
arekinath1
rip
-
arekinath1
jk it still runs fine
-
arekinath1
there's always minimanta
-
Agnar_
minimanta?
-
arekinath1
my shitty mostly-manta-API-compatible thing which stores the data as plain files on disk and puts the metadata in the resource fork
-
arekinath1
basically if you want to be able to use the manta tools and msign etc against just a plain directory on a single box somewhere, it's that
-
dlg
should we donate capacity?
-
arekinath1
our capacity is in .au though
-
arekinath1
slow for everyone except us haha
-
dlg
anything free is worth what you pay for it
-
dlg
and then we'll find out these vmdumps are really miners
-
arekinath1
well if LeftWing (IRC) wants a manta on the moon to store shiz in for a dump upload service he knows where to find us
-
dlg
yep
-
LeftWing
You guys should put up some project SPARC boxes in your outback data cupboard
-
arekinath1
I guess 200ms away is not long enough for the moon, but it's further than geostationary orbit at least
-
LeftWing
SPARC boxes can't respond to an SSH packet in less than 200ms anyway
-
dlg
LeftWing: i do have a serious offer from a friend in alice springs to host gear
-
LeftWing
Haha think of the dust
-
» dlg cover the ears of his m3000
-
LeftWing
I wonder if we would boot on one of those
-
arekinath1
I did seriously consider bringing some Cool Threads home with me
-
Agnar_
LeftWing: good news on sparc btw. I got the information that the T4s got free, the applications got migrated. Now we wait for the official "ok" to shut them down. I'll get three T4-4, 2sockets each, 256G RAM each. I'll make two out of three and I plan to bring the resulting 4 socket, 512G one into a proper datacenter
-
dlg
there was code for them
-
LeftWing
arekinath1: Because you didn't take them home, Samsung destroyed them
-
arekinath1
yeah well our bags were too full as it was
-
arekinath1
just ask the qantas people who had to help us with them
-
LeftWing
Ha
-
dlg
dlg@opiate ~$ /usr/bin/time ssh 130.102.96.36 sysctl hw.product
-
dlg
hw.product=SUNW,SPARC-Enterprise
-
dlg
0.65 real 0.01 user 0.00 sys
-
dlg
grumble
-
dlg
i blame pivy for most of that time
-
arekinath1
lol
-
arekinath1
yk5 ec signature takes about 70-80ms
-
arekinath1
you using rsa?
-
dlg
how dare you
-
dlg
debug1: kex: algorithm: curve25519-sha256
-
dlg
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
-
dlg
debug1: kex: server->client cipher: chacha20-poly1305⊙oc MAC: <implicit> compression: none
-
arekinath1
even rsa2048 is only about 140ms on a yk5, and that's with all the pivy-agent overhead in ;)
-
dlg
using an ed25519 key on disk is about 0.45 real
-
dlg
ANYWAY
-
dlg
sparc is fine
-
dlg
time to run that same command is about the same this amd seattle box
-
arekinath1
not slow at all
-
arekinath1
yeah ssh to the machine next to the one I'm typing on often takes 200ms
-
arekinath1
it has to do a bunch of roundtrips and thinking and then start a pty and shell
-
arekinath1
bunch of work
-
LeftWing
The control socket advantage is real
-
arekinath1
uh huh
-
LeftWing
At least for establishment
-
dlg
true
-
wilbury
LeftWing: re dump upload: sort of "supportfiles.sun.com"?
-
Agnar_
LeftWing: btw, I was about to write a kind of a generic "explorer" for illumos systems...do you see a need for that?
-
nanxiao
Just curious: who owns this account:
twitter.com/Illumos_org?
-
sjorge
nanxiao: LeftWing I think
-
sjorge
based on his demand for followers :p
-
sjorge
Smithx10: are those NVMe's in a mirror?
-
sjorge
If so, you could probably zpool offline them before flashing?
-
sjorge
Thats what I did when I had to move some disks around in a chassis, offline it, pull it, plug it, online it, wait for resilver, repeat
-
sjorge
scrub
-
andyf
sjorge: @illumosorg is the new one that LeftWing mentioned, @Illumos_org (note with the incorrect capital I) - no idea
-
sjorge
didn't even notice the _
-
sjorge
maybe we could get the one from leftwing the verified checkmark
-
sjorge
somehow
-
Smithx10
sjorge: sadly these are in raidz
-
sjorge
Smithx10 I suppose technically you can offline these too but... hmmm... seems more risky than mirror to me
-
Smithx10
sjorge: I'm not sure, but... it looked like you needed to power cycle to the device to get it to use the firmware in the new slot
-
Smithx10
not sure how it is supposed to behave
-
jimklimov
got friends asking about a Solaris 10u11 system, it fails to mount a ZFS pool from an iSCSI NAS after some outage on that NAS. It was even zfs-mirrored over two volumes from that NAS, and with a separate log-device mirrored, but but when `zpool import` processes it (even in `-F -n` mode to try rolling back), the kernel panics
-
jimklimov
assertion failed: 0 == dsl_dataset_hold_obj(<args>) (0x0 == 0x32), file: .../dsl_scan.c, line: 1359
-
tsoome
jimklimov I'd use 11.4 media
-
jimklimov
I suppose I can try to walk the pool halves with zdb to see if any is okay, but not sure over the years which commands can do effectively the userspace equivalent of import and show what is the error without a kernel panic
-
jimklimov
or maybe import without the log devices, but no idea how to check if they are the problem ")
-
jimklimov
or a new distro indeed?
-
jimklimov
do you mean booting with new live media and trying to import?
-
jimklimov
can live media be an iscsi client? :)
-
igork
prepare new media with client :)
-
tsoome
well, the point behind 11.4 is simple... you get no patches for 10 and this kind of assert smells for patch.
-
tsoome
there is a price for running ancient software.
-
jbk
tsoome: aside from the bootparams that are passed to the kernel, is there any other mechanism of passing data to the kernel?
-
tsoome
loader env is
-
rmustacc
And there are more general multiboot tags.
-
rmustacc
There's a lot of data that gets to dboot/unix, but they have to decide to do something with it.
-
tsoome
I have used just module for loader env, and console font
-
andyf
Agnar_: I think I have finally replicated your postgres HEAD hang..
-
andyf
Agnar_: which process was it that caused the panic when you killed it? server or client
-
sean____
Heya, extremely noob question here, but I saw
illumos.org/books/dev/workflow.html…r-best-friend-while-building-bldenv has a link to the bldenv man page, but that link appears broken
-
sean____
I can tell that bldenv starts a new shell, and that it helps me do incremental builds (somehow?) but is there something I can read for more detail on what it's actually doing?
-
wilbury
sean____: yes. use usr/src/tools/scripts/bldenv
-
jbk
bldenv mostly just sets a _lot_ of environment variables for the shell it starts
-
jbk
stuff that the build will want/need
-
wilbury
sean____: nightly -i is perhaps what you need
-
jbk
i would _strongly_ suggest doing a full build before attempting any sort of incremental build
-
sean____
jbk: Yup, I did (and booted into the new BE successfully), now trying to work through the incremental flow
-
jbk
there are unfortunately more rough edges around incremental builds than i think any of us would like (never enough hours in the day)... so just be prepared
-
jbk
(sometimes it requires a bit more knowledge of the build internals to get it to do the right thing when incrementally building, which ideally shouldn't be necessary)
-
jbk
but we can try to help sort through that too
-
sean____
heh I'm not sure I've ever worked with a system that hasn't had rough edges around incremental rebuilds in *some* cases, so I'll mentally prep myself :)
-
jbk
i always find it's good to set expectations.. and to make sure you know help is available :)
-
jbk
tsoome: i was thinking about ways we could pass a key to the kernel from the bootloader for an encrypted root pool (granted I suspect the boot files in the pool would still need to be on unencrypted datasets)
-
tsoome
atm not just the boot files but also /platform
-
tsoome
IMO there are 2 possible ways, use the normal module approach with specific name, as we do for env/font; or invent special structure and push it to specification.
-
tsoome
both are doable
-
jbk
does the bootloader construct those modules dynamically?
-
tsoome
some sort of structured approach is needed anyhow, but, now we can encode nvlists in loader
-
tsoome
yes
-
jbk
ahh ok
-
tsoome
in common/module.c build_font_module() and build_environment_module()
-
tsoome
module approach is handy because we already have code in kernel to release modules, only need to add secure memory wipe too:)
-
jbk
i've been trying to get details on how linux or windows handles whole disk encryption of their boot disks -- as best as i can tell, there's some small unencrypted partition that it starts from
-
tsoome
for zfs, we have 2 cases, loader is loaded from ESP (UEFI), it is not encrypted, loader can ask for pass/key and go on. And for x86 we have gptzfsboot in pool label area, which is also not encrypted.
-
tsoome
so that part is already good
-
tsoome
just need to add decrypt for loader zfs reader.
-
tsoome
+ key transport and pickup mechanism for kernel.
-
jbk
i'm surprised there hasn't been any work on the fbsd for this.. seems like something someone would have jumped on by now :)
-
jbk
err fbsd side of the loader
-
tsoome
I have it in my queue:D
-
tsoome
in todo queue, that is.
-
tsoome
There are not too many candidates who might pick the task:D
-
tsoome
and most are busy preparing 13 for now
-
tsoome
.oO and I am writing instructions for a customer, how to migrate and decommission old brocade FC switches:D
-
ypankov
tsoome: `pkg update` doesn't take care of updating the ESP bits?
-
tsoome
but anyhow, ability to read encrypted pools is almost in top of my list
-
tsoome
ypankov it should do
-
tsoome
but it depensd on how old is the system
-
tsoome
"it" in this context means relatively up to date system
-
tsoome
beadm activate also should.
-
tsoome
both are using the libbe.so for that task anyhow
-
ypankov
ah, could it be that 7357 did not update the usr/src/boot/Makefile.version? (still fighting with that change :D)
-
Agnar_
andyf: it hangs on the client and if I kill them, the system panics
-
tsoome
ypankov hm? git diff...
-
ypankov
i mean, if Makefile.version was not updated, libbe does not update ESP?
-
tsoome
yes, the update is based on version number
-
ypankov
yep, increasing that version made it update ESP, thanks
-
tsoome
was it missing?
-
ypankov
yep, as i said, 7357 didn't update the version (and i'm merging upto that change)
-
ypankov
it's probably fine with all the updates that were after that
-
tsoome
7537 :D but, you also want 13345 if there are vdevs removed from pool...
-
gitomat
[illumos-gate] 13628 sppptun: array subscript 0 is outside array bounds -- Toomas Soome <tsoome⊙mc>
-
ypankov
right, i can't spell
-
ypankov
tsoome: any plans on efibootmgr-like implementation?
-
tsoome
eventually
-
tsoome
damn, another Fast Data Access MMU Miss. time to walk my dog:)
-
gitomat
[illumos-gate] 13697 zfs change-key does not follow clones, data loss ensues -- Tom Caputi <tcaputi⊙dc>
-
gitomat
[illumos-gate] 13601 ksh shell lint misleading -- Andy Fiddaman <omnios⊙ccu>
-
jimklimov
-
jimklimov
Our previous generation of database servers had dual Intel Xeon E5-2650 v4 CPUs, 24 physical cores in total. They had 1TB memory with 24 3.8TB SSDs connected via SATA in a RAID 10 configuration. These worked fine for daily issuance but would not be able to handle replacing all of our certificates in a single day. We have replaced them with a new generation of database server from Dell featuring dual AMD EPYC 7542 CPUs, 64 ph
-
jimklimov
cores in total.
-
jimklimov
These machines have 2TB of faster RAM. Much faster CPUs and double the memory is great, but the really interesting thing about these machines is that the EPYC CPUs provide 128 PCIe4 lanes each. This means we could pack in 24 6.4TB NVME drives for massive I/O performance. There is no viable hardware RAID for NVME, so we’ve switched to ZFS to provide the data protection we need.
-
andyf
Nice :)
-
andyf
Dell was always a problem because of the RAID cards (unless you could get 'Nexenta spec'), but with NVMe it's much nicer
-
wilbury
i've deployed a big'ol zfs storage with dell r740xd recently. 2xSSD, 22xNVME, freebsd13+zfs
-
wilbury
approx 25TB raw capacirty
-
andyf
-
andyf
I don't understand why, if neither FSEARCH nor FEXEC are set, it basically sets loads of bits on
-
tsoome
directory permissions?
-
jbk
maybe (just guessing) since it's recording the open files, the idea is distinguishing between mapped and just open for read/write?
-
tsoome
you can open file (by name) when you can not search the directory content
-
rmustacc
No, this is something a bit weird.
-
rmustacc
It adding it seems suspect, but the prolem is that there's basically two different sets of flags in the file_t.
-
rmustacc
Look at f_flag an f_flag2.
-
rmustacc
And there's a bunch of things around how FSEARCH|FEXEC deal with that pattern.
-
andyf
Yes, flag2 being the 'open only' ones
-
rmustacc
Not saying I undersatnd it, but it's usually related to that.
-
andyf
and the bottom few bits of open() flags are mutually exclusive
-
rmustacc
Whether that should be += or &= I'm not sure.
-
andyf
but for some reason, when dumping the info to a core file, it ends up adding 0xffffffff if neither flag is et
-
andyf
but for some reason, when dumping the info to a core file, it ends up adding 0xffffffff if neither flag is set
-
andyf
and yes, += seemed odd too
-
andyf
(the same thing happens in /proc/xxx/fdinfo, but that's because I based that on this stuff
-
rmustacc
Though this is what F_GETFL does.
-
rmustacc
The += would cause things to overflow if bits aren't set. But I don't know enough of the context here.
-
andyf
ah right, it's in F_GETFL too
-
rmustacc
It'll require a bit of archaeology to figure out, but I suspect raf had a plan.
-
jimklimov
regarding my earlier post, here's HW-centric details (and improvement graphs) of LE current generation of 24xNVMe EPYC servers:
letsencrypt.org/2021/01/21/next-gen-database-servers.html and their setup overview from getting feet wet with OpenZFS:
github.com/letsencrypt/openzfs-nvme-databases
-
andyf
rmustacc - ok, thanks - at least I am not missing anything obvious.
-
andyf
it has an interesting affect on `pfiles` output - lots of flags set
-
andyf
*effect
-
rmustacc
andyf: Yeah. I'm not quite sure there. We'll have to dig.
-
rmustacc
It does seem suspect on pfiles.
-
andyf
well, same with `elfdump -n` on a core
-
andyf
but I suppose that information is there for `pfiles` ultimately..
-
rmustacc
We'll just need to come to an understanding of that and leave some comments for future us.
-
sean____
Is there a recommended gitignore for illumos-gate? after running the build, "git status" is returning hundreds of new files in varying locations with many different suffixes
-
LeftWing
sean____: I think folks tend to "git config status.showUntrackedFiles no"
-
LeftWing
Though it would certainly be good to have a .gitignore that was kept up-to-date
-
sean____
This is gonna sound stupid, but I would expect that to cause problems for people sending commits for review, and accidentally leaving things out - does that not happen in practice?
-
LeftWing
It would be a lie to say it has never happened haha
-
LeftWing
We try pretty hard to make sure a full clean build is done before integrating a change