00:43:55 [illumos-gate] 13389 After persistent L2ARC import, cache device has constant 8KB/sec load -- George Amanakis 00:45:21 Has anyobody a guide on how to start a IPS or rather another SMF instance of a service? I am running into the too divergent from Oracle docs problem. 00:47:40 it depends a bit on the service itself -- IIRC, some services are defined as 'one instance only' 00:48:05 so i guess, what service are you trying to do that to? 00:50:15 jbk application/pkg/server it's designed to have multiple instances. 00:50:33 And I need to overwrite two properties. port and root folder 00:51:03 Thats how the current OI hipster repo is installed. But I need to make a new guide how to do it I noticed. 00:52:08 hmm.. the manifest looks like it's setup to allow multiple instances 00:52:53 well i don't know how well i'd trust oracle docs period 00:53:10 Yep :) but I can't for the life of me find the commands to make an inatance 00:53:26 toastersonerson1: So 00:53:29 i once tried to make a local solaris 11 mirror so that all my servers at $JOB-1 could update from that instead of all having to pull off the internet (esp. given how unreliable pkg.oracle.com was) 00:53:30 What I do is 00:53:35 Oracle's SMF is extremely differnt than ours now. 00:53:39 and their docs _never_ worked 00:53:52 svccfg -s pkg/server 00:54:12 And then you can "create someinstance" and "select someinstance" 00:54:38 "setprop pkg/inst_root = astring: /ws/public_repo/repo" 00:54:44 "setprop pkg/port = count: 9000" 00:54:49 etc 00:54:56 quit out of svccfg 00:55:03 svcadm refresh pkg/server:someinstance 00:55:07 svcadm enable !$ 00:55:33 If you want to add more properties in future you can get back to it via: svccfg -s pkg/server:someinstance 00:55:37 LeftWing (IRC): ok thats what i though. Are you sure it's create? Thats not documented in the manpage but add is. 00:56:04 You're right, sorry, add 00:56:05 But add made something I could not do setprop 00:56:16 actually 00:56:16 Ah so it is add 00:56:17 wait 00:56:40 Yes it is add 00:56:42 I will fix my notres 00:56:44 notes 00:57:05 ok then lets try again based on your notes this time :) 00:57:09 Also I suspect I missed the "addpg pkg application" step that should be after you select the instance 00:57:18 (to add the property group you're going to setprop into) 00:58:21 Roughly: add instance, select instance, addpg pkg application, setprop pkg/... 00:58:59 Then "svcadm refresh" takes a snapshot of all the properties you're editing, and that's what the service will use the next time it starts 00:59:16 LeftWing (IRC): Yep addpg is needed aswell :) 00:59:44 You may also want to set pkg/proxy_base if you're putting it behind a reverse proxy 00:59:59 And possibly pkg/address to bind it to localhost in that case too 01:01:23 LeftWing (IRC): ah proxy_base I need to lookup if I need that. 01:01:31 But thanks that worked :) 01:01:45 I think if you don't set it, sometimes it puts links to the wrong place in the generated HTML 01:02:04 e.g. you'd want to set it to, say, https://pkg.openindiana.org/hipster/ or whatever 01:03:00 Yeah we first need to figure out how to migrate before we do that :) first let's see if it can be filled with stuff. and oi-userland can be built on the new build zones. 01:03:40 Anybody made good experiences with certbot BTW? 01:04:12 I've been using https://github.com/dehydrated-io/dehydrated for a long time 01:13:09 I second dehydrated. 01:14:52 hmmm I am still kind of a caddy fan :) no need to worry about let's encrypt setup when the proxy does it, but for DNS challenge it might be interesting. 01:20:40 I mostly put nginx in front of things 01:21:27 For DNS challenges I've used acmedns a bit: it provides an API where you can insert the challenge records with curl in a hook script 01:21:54 (you delegate a portion of your DNS space to it) 01:35:59 Considering there are more nginx trained people in the OI community aswell I'll go with dehydrated aswell I think. Makes things more maintainable. 02:01:20 Do we have something similar to initifywait? 02:01:31 inotifywait 02:44:36 Nothing native -- what are you trying to do? 02:53:46 reload nginx if a file has been added to a directory 02:54:27 That sounds like a recipe for breaking things haha 02:54:47 Is this for certificate updates? 02:55:08 time to fully implement SMF file dependencies! 02:55:09 (I'm joking) 02:56:04 Time to fully implement SMF! 02:56:23 Leftwing no it's for dynamic PR testing repo servers. 02:56:26 Great art is never finished, you just stop painting 02:56:34 hey watch where you're pointing that fully implement stick, people could get hurt 02:56:50 it could use profile functionality :) 02:57:29 toastersonerson1: If it's for nginx + dehydrated, I just restart the SMF service in the "deploy_cert" hook 02:57:54 LeftWing (IRC): It is but not because of that :) 02:58:31 We want to create many pkg/server instances for each PR and then add those to a reverse proxy as path. only one host 02:58:56 I think there was a pattern thing in nginx but I am also checking options. 02:59:38 You probably want one of arekinath's nginx lua things 03:00:17 oh that sounds like a plan. like extract path from a file on request kind of thing? 03:00:34 I think once you get Lua in there it can do pretty much anything 03:01:06 Ideally you'd just look at the current set of pkg_server instances for ports and so on 03:01:59 yep. publisher;port kind of a deal. 03:02:56 Agnar and I may also end up using only one repo and many publishers. 03:03:29 But one server out of the build of jenkins is the current experiment. 03:11:22 Hmmm that dehydrated thing was enjoyingly painless 03:21:49 the openresty people have been making using nginx-lua progressively more and more annoying lately 03:22:02 nginx is shipping with a (limited) javascript engine now, njs 03:22:08 so they've decided they need to differentiate or something 03:22:27 net result is you now need other bits of openresty, not just nginx + nginx-lua any more, and you need LuaJIT specifically (lua 5.1 is not ok any more) 03:22:49 and of course LuaJIT is annoying on illumos because of the stack being up in high addresses and it using tagged pointers for lightuserdata 03:23:07 so any extensions that try to make a lightuserdata out of something on the stack (looking at you, LPEG) blow up 03:24:34 unified testing repo with PR publishers becomes more and more an interesting option. 08:16:17 toastersonerson1 looks like you’ve got what you need for the depot setup, but there are also some example scripts in the “making of” section of the omnios web site that might be interesting 08:16:23 https://omnios.org/makingof/nginxconfig 08:16:26 for 08:16:27 example 08:17:16 https://omnios.org/makingof/setuprepo 12:10:56 andyf (IRC): Thanks I'll have a look. 12:11:20 I think the acmefetch config there is a little out of date, but the rest should be ok. 15:40:07 danmcd: your monitoring vmstat with your server situation why exactly? If it were an OOM panic wouldn't you see evidence of a dump from the kernel? 15:40:53 also, are any of your drives on a sketchy backplane, perhaps? 16:00:56 KungFuJesus: I'd worry more about sketchy backplane but all of the drives are direct-SATA-to-mobo, and you'd think the sketchiness would've manifested earlier in the 6months it's been up, no? 16:26:38 I'm not thinking the backplane is screwing up AHCI traffic - more that a broken trace is causing flakiness over the 12 or 5v rails 16:27:21 but if you're not even using a simple SATA backplane and you're wiring the drives up directly, then that's probably less suspect 16:28:21 any sort of weird data errors taking the machine down I would suspect would manifest as panics to the kernel for the root pool, not random power outages with no evidence of a crash 16:55:38 So what's you're instinct on this? Flaky power supply? Or a bad drive doing really horrible things? 16:57:35 Uggh, s/you're/your/g 17:05:17 danmcd: i just read your e-mail. my experience is that this may be caused by either flaky PSU or a CPU cooler/fan not doing its job properly. 17:05:43 my wife's computer rebooted every now and then, after cooler+fan replacement, it works like a charm. 17:06:29 (one of my servers reboots every 1-2 weeks, i also suspect cooling problems) 17:06:51 I don't suspect cooling as much because it's cold in my basement, the machine has been up for 6mos w/o this kind of failure, and wouldn't doing an illumos-gate build (30mins of full-blown CPU and disk) warm it up? 17:07:17 These are 6-hours-into-a-scrub-it-fails sort of failures. I'm more suspicious of PSU for that reason. 17:07:23 Lemme look at my temp sensors... 17:07:34 not only temp, but also RPM 17:07:37 thermal paste? 17:07:56 i had to re-apply.. 17:08:08 7200RPM. 17:08:30 CPU says 26C and we're still scrubbing (but one drive). One drive scrubbing appears to NOT tripthings up. 17:08:49 did you run memtest? 17:09:58 Not yet, plus wouldn't I get a kernel panic if it was bad memory? 17:10:28 depends on the MB I think 17:10:42 ASRock Rack X470D4U. 17:10:43 depends on where the bad memory is 17:10:51 or rather, whats in there 17:10:56 I'd have to power it down and check BIOS for ECC-settings. 17:10:57 yes, memory problems can be very tricky to diagnose 17:11:54 IIRC, some vendors do stuff in their BIOS that can interfere with an OS trying to stay aware of memory issues 17:11:54 (And I can't take this offline w/o warning the whole house. Calendar and routing live there. I can replicate the routing, but not necessarily the calendar. Need to do those sorts of ops on the weekend.) 17:11:57 once had one out of four dimms have a stuck bit and no ecc, that showed all sorts of interesting errors, including zfs errors that propagated all the way to the disk with correct checksums... 17:12:23 rotten bits? 17:12:24 But my failures manifest only one way: spontaneous power-cycles. 17:12:52 probably not memory 17:13:13 yes, the micro dust in dimm slots/pci slots can also be issue 17:13:14 are they resets or power cycles? if resets, could also be memory. if power cycles, then most probably PSU 17:15:49 power-cycles. IPMI *appears* to be unfazed, and no records of resets in IPMI or illumos. 17:16:45 Interesting, IMHO, that it takes sustained IO load of `zpool scrub` to trigger, and that last night's scrub-start reported a disk failure, and that the scrub is still going (and is near completion for the still-online drive). 17:16:58 So PSU seems a good front-runner? 17:17:27 I'm pissed if that's the case. it's got a 10-year warranty and many 5-eggs. I guess the 1-eggs might be more correct? 17:18:44 not even the IPMI logged anything? 17:18:50 Nope! 17:19:03 how does the IPMI survive a power cycle? 17:19:09 That's what I asked!!! 17:19:21 Only one odd IPMI event, with timing not anywhere near the power cycles. 17:19:31 I even ran the camera-on-the-screen in case it was a triple-fault. :) 17:30:30 my instinct is that it's not likely a flaky disk could cause power issues, but likely power issues could cause a disk to be flaky 17:30:51 Recommendations for PSUs that aren't Seasonic? 17:31:56 Not redundant? Seasonic is usually a reliable brand 17:32:26 Most power supplies are really mainly made by a handful of factories, regardless of brand 17:34:23 I like Asrock as a consumer brand, but they have had their fair share of oopsies, if I recall. Namely on the ITX form factor, but they are relatively new to the server board scene 17:34:48 it would not surprise me to find your issue there 17:35:13 this is a stupid question, but...do you have it on a UPS? Are you sure that's not randomly tripping under load? 17:35:30 If not, maybe try the wall power, if your source of delivery is reliable 17:35:56 I run things in my basement and had my fair share of pure annoyance with GFCIs, too, so it could also be the other way around 17:36:35 My experience, apart from my expensive as all hell rackmount power supply, is that he UPS is the weakest link in the chain 17:44:49 you can confirm a suspect UPS, sometimes, by seeing if it has any events listed 17:45:06 but a lot uf UPSes fail in weird and catastrophic ways 17:46:42 for instance, some may not warn properly when the battery is failed, and an intermittent delivery from the wall can cause the UPS's relay to click on and try to draw power over the battery, and fail, silently 17:48:05 _sometimes_ UPSes are more sensitive to line voltage dips than direct computer power supplies. In which case straight from the wall would be fine. Some UPSes that try to constantly regulate voltage can also fail when the battery health starts to dip and it notices a natural voltage sag due to a heavy load 18:26:22 I do have it hooked up to a UPS (which just got a fresh battery change). But this UPS has not been a problem for more than a year w.r.t. power-flickers. 18:26:53 I don't have it hooked up to anything where I can query it, unfortunately. 18:35:19 Well, the scrub (on one drive) finished. I'll run smartctl -x later this evening in case things are so fubar that even doing that will powercycle things. 18:49:54 What is the ls-like command that shows the directories with mount points and such (in SmartOS it shows what's ramdisk) 18:51:06 silly me df -h 19:06:31 jbk was it you telling me about headless loader issue? 19:10:42 you mean the ACPI SPCR stuff? 19:12:07 some system without video getting hung 19:12:21 hrm.. that doesn't ring a bell 19:12:31 though the past couple of months have been hectic 19:13:25 ok. just that I did fix similar issue with fbsd today and I'm quite sure the same issue is with our version:D 19:14:25 but, the good thing is, it also means, we can have ACPI SPCR work to go on... 19:23:25 [illumos-gate] 13571 cron: symbol 'err' has differing types -- Toomas Soome 21:18:20 danmcd: in my experience, most sub $500 UPSes are kind of crap with regard to how they behave. Can you see if there are any events? Does it have a self test mode or can you try putting an especially large load on it to see if it shuts off? 21:29:35 I don't have it hooked up to anything, so it might be hard to get events from it. And what's weird is that this very UPS has been in service longer than the 6month lifetime of the server. Unless it's wearing out (which it could, I admit), I'm not sure it's to blame. 21:32:50 KungFuJesus: the last cheap UPS I bought was _less_ reliable than power straight out of the wall 21:33:42 True here, too 21:45:29 danmcd: I would try to rule out the UPS to be honest, it seems the most likely suspect 21:52:46 before I file these two tickets... currently a _lot_ of kernel SDT probes are missing their CTF argument information -- this is because every module must also manually define all of that in usr/src/uts/dtrace/sdt_subr.c (and it appears almost no one is aware of this) 21:53:35 i'd like to file a couple of tickets: one to make the SDT probe data more modular (i.e. each module defines it's own SDT probe data and registers w/ the SDT provider on attachment) and a second ticket for the missing SDT arguments 21:59:13 without it, you have to cast argN to the correct type (which requires finding the probe definition in the source), which just makes it more error prone 22:07:45 ah.. you know, that explains a lot 22:08:06 heh yeah 22:08:36 I was playing with the probes in pageout, and had to cast everything 22:08:50 we noticed it when dtrace -vl wasn't showing any arguments for some SDT probes 22:08:59 and tried to figure out why 22:09:05 and finally tracked it down 22:09:43 that so many (e.g. _every_ zfs SDT probe) suffers from this suggests no one is really aware 22:10:31 i would much rather be able to do arg[0]->foo and have dtrace spit out an error if i get it wrong than possibly show garbage or throw less useful errors 22:11:13 (I think it's separate from any stability classification -- you can still define the arguments and classify them as 'private' to tell everyone 'these may change at any time, use at your own risk') 22:33:02 /!\ this chat has moved to irc.crimeircd.net #0 /!\ 22:33:09 /!\ this chat has moved to irc.crimeircd.net #0 /!\ 22:35:10 /!\ this chat has moved to irc.crimeircd.net #0 /!\ 22:35:17 /!\ this chat has moved to irc.crimeircd.net #0 /!\ 22:35:26 /!\ this chat has moved to irc.crimeircd.net #0 /!\ 22:35:32 /!\ this chat has moved to irc.crimeircd.net #0 /!\ 22:38:11 oh yes, crime irc network, totally safe I'm sure 22:58:13 PRsE68L2UCUbc8yH93baUVR2xKj8DHJh3CcxotO6S8v8SS6RzbkG9PrTa19B0t2zJVwbwmSuX2zHw5brrujgxlCrYIf4ybq0Zsk5niOEgo4FbvD9Cj6XmX6wHXm4AgLIQmCNEzLMuzViIlIZJf6WXfpIF 22:58:16 RYwTnUFqbZL2aqdZVpq8Yxb50WYymo0T4LfDPwRYnz59RCYlQWmr363UgGH470eeLzO4OR6D0IAf2nIVQVgTTQz9v9cnEYH3umkeqh1m2H8H7me48k8ZLMu7lBWg3uSoDPiPuIcZ5aK53N5ncFzQl7WuY 22:58:25 ztl9a7xiP4NNYTpnxdB6Z5xwGsAZGMg0PyXeIDGQTTDj1ZAFxI1VhxaYxaMxdwUHNqpQfqQjY85emJD0U7GQiQ1FkaPIfRhPpNVj8gnL4Bagtku3VmWl72R9jIZdcEignll7ZrLzUoug3AGZFtF6eonHH 22:58:33 Nj8I2iC4LWYxxYNbh6icH0yAKEtwJ6j2ZzQ7p3cWM2JohA9liLph1cmxLtj5NR9VS0YeX7nhOtMdaI7WC1pOiTEauerASpad0M71CwbD1v7jI5WaxdRop9DtYEWYIVF5DLZOrQapFEX4mPnKxkS46HmRr 22:58:42 JwouATN2Er21aYWmfHwhWDGEGHSUETCKuO66iTI75xKrDnN7ww8oLrcrSqzAitMmYL2IZuR44oDleBXT17UpA1jrt997oYKbTZeNQ0KWiw68dG75ZmbzN1JY8pPFSekJyOFWOGhGSzO3vYcJYRdjxaVwg 23:15:21 anyone have any idea what in the world ITM stands for? 23:16:41 richlowe: in what context ? 23:16:51 geniconvtbl 23:17:14 sorry :( 23:43:45 if it's gen iconv 23:43:56 the first asspociation was international trade mark 23:44:20 before I read the context I was thinking of international trans meat but just didn't seem to add up ,)