-
gitomat
[illumos-gate] 13389 After persistent L2ARC import, cache device has constant 8KB/sec load -- George Amanakis <gamanakis⊙gc>
-
toastersonerson1
Has anyobody a guide on how to start a IPS or rather another SMF instance of a service? I am running into the too divergent from Oracle docs problem.
-
jbk
it depends a bit on the service itself -- IIRC, some services are defined as 'one instance only'
-
jbk
so i guess, what service are you trying to do that to?
-
toastersonerson1
jbk application/pkg/server it's designed to have multiple instances.
-
toastersonerson1
And I need to overwrite two properties. port and root folder
-
toastersonerson1
Thats how the current OI hipster repo is installed. But I need to make a new guide how to do it I noticed.
-
jbk
hmm.. the manifest looks like it's setup to allow multiple instances
-
jbk
well i don't know how well i'd trust oracle docs period
-
toastersonerson1
Yep :) but I can't for the life of me find the commands to make an inatance
-
LeftWing
toastersonerson1: So
-
jbk
i once tried to make a local solaris 11 mirror so that all my servers at $JOB-1 could update from that instead of all having to pull off the internet (esp. given how unreliable pkg.oracle.com was)
-
LeftWing
What I do is
-
toastersonerson1
Oracle's SMF is extremely differnt than ours now.
-
jbk
and their docs _never_ worked
-
LeftWing
svccfg -s pkg/server
-
LeftWing
And then you can "create someinstance" and "select someinstance"
-
LeftWing
"setprop pkg/inst_root = astring: /ws/public_repo/repo"
-
LeftWing
"setprop pkg/port = count: 9000"
-
LeftWing
etc
-
LeftWing
quit out of svccfg
-
LeftWing
svcadm refresh pkg/server:someinstance
-
LeftWing
svcadm enable !$
-
LeftWing
If you want to add more properties in future you can get back to it via: svccfg -s pkg/server:someinstance
-
toastersonerson1
LeftWing (IRC): ok thats what i though. Are you sure it's create? Thats not documented in the manpage but add is.
-
LeftWing
You're right, sorry, add
-
toastersonerson1
But add made something I could not do setprop
-
LeftWing
actually
-
toastersonerson1
Ah so it is add
-
LeftWing
wait
-
LeftWing
Yes it is add
-
LeftWing
I will fix my notres
-
LeftWing
notes
-
toastersonerson1
ok then lets try again based on your notes this time :)
-
LeftWing
Also I suspect I missed the "addpg pkg application" step that should be after you select the instance
-
LeftWing
(to add the property group you're going to setprop into)
-
LeftWing
Roughly: add instance, select instance, addpg pkg application, setprop pkg/...
-
LeftWing
Then "svcadm refresh" takes a snapshot of all the properties you're editing, and that's what the service will use the next time it starts
-
toastersonerson1
LeftWing (IRC): Yep addpg is needed aswell :)
-
LeftWing
You may also want to set pkg/proxy_base if you're putting it behind a reverse proxy
-
LeftWing
And possibly pkg/address to bind it to localhost in that case too
-
toastersonerson1
LeftWing (IRC): ah proxy_base I need to lookup if I need that.
-
toastersonerson1
But thanks that worked :)
-
LeftWing
I think if you don't set it, sometimes it puts links to the wrong place in the generated HTML
-
LeftWing
e.g. you'd want to set it to, say,
pkg.openindiana.org/hipster or whatever
-
toastersonerson1
Yeah we first need to figure out how to migrate before we do that :) first let's see if it can be filled with stuff. and oi-userland can be built on the new build zones.
-
toastersonerson1
Anybody made good experiences with certbot BTW?
-
LeftWing
-
danmcd
I second dehydrated.
-
toastersonerson1
hmmm I am still kind of a caddy fan :) no need to worry about let's encrypt setup when the proxy does it, but for DNS challenge it might be interesting.
-
LeftWing
I mostly put nginx in front of things
-
LeftWing
For DNS challenges I've used acmedns a bit: it provides an API where you can insert the challenge records with curl in a hook script
-
LeftWing
(you delegate a portion of your DNS space to it)
-
toastersonerson1
Considering there are more nginx trained people in the OI community aswell I'll go with dehydrated aswell I think. Makes things more maintainable.
-
toastersonerson1
Do we have something similar to initifywait?
-
toastersonerson1
inotifywait
-
LeftWing
Nothing native -- what are you trying to do?
-
toastersonerson1
reload nginx if a file has been added to a directory
-
LeftWing
That sounds like a recipe for breaking things haha
-
LeftWing
Is this for certificate updates?
-
richlowe
time to fully implement SMF file dependencies!
-
richlowe
(I'm joking)
-
toastersonerson1
Time to fully implement SMF!
-
toastersonerson1
Leftwing no it's for dynamic PR testing repo servers.
-
LeftWing
Great art is never finished, you just stop painting
-
arekinath
hey watch where you're pointing that fully implement stick, people could get hurt
-
toastersonerson1
it could use profile functionality :)
-
LeftWing
toastersonerson1: If it's for nginx + dehydrated, I just restart the SMF service in the "deploy_cert" hook
-
toastersonerson1
LeftWing (IRC): It is but not because of that :)
-
toastersonerson1
We want to create many pkg/server instances for each PR and then add those to a reverse proxy as path. only one host
-
toastersonerson1
I think there was a pattern thing in nginx but I am also checking options.
-
LeftWing
You probably want one of arekinath's nginx lua things
-
toastersonerson1
oh that sounds like a plan. like extract path from a file on request kind of thing?
-
LeftWing
I think once you get Lua in there it can do pretty much anything
-
LeftWing
Ideally you'd just look at the current set of pkg_server instances for ports and so on
-
toastersonerson1
yep. publisher;port kind of a deal.
-
toastersonerson1
Agnar and I may also end up using only one repo and many publishers.
-
toastersonerson1
But one server out of the build of jenkins is the current experiment.
-
toastersonerson1
Hmmm that dehydrated thing was enjoyingly painless
-
arekinath
the openresty people have been making using nginx-lua progressively more and more annoying lately
-
arekinath
nginx is shipping with a (limited) javascript engine now, njs
-
arekinath
so they've decided they need to differentiate or something
-
arekinath
net result is you now need other bits of openresty, not just nginx + nginx-lua any more, and you need LuaJIT specifically (lua 5.1 is not ok any more)
-
arekinath
and of course LuaJIT is annoying on illumos because of the stack being up in high addresses and it using tagged pointers for lightuserdata
-
arekinath
so any extensions that try to make a lightuserdata out of something on the stack (looking at you, LPEG) blow up
-
toastersonerson1
unified testing repo with PR publishers becomes more and more an interesting option.
-
andyf
toastersonerson1 looks like you’ve got what you need for the depot setup, but there are also some example scripts in the “making of” section of the omnios web site that might be interesting
-
andyf
-
andyf
for
-
andyf
example
-
andyf
-
toastersonerson1
andyf (IRC): Thanks I'll have a look.
-
andyf
I think the acmefetch config there is a little out of date, but the rest should be ok.
-
KungFuJesus
danmcd: your monitoring vmstat with your server situation why exactly? If it were an OOM panic wouldn't you see evidence of a dump from the kernel?
-
KungFuJesus
also, are any of your drives on a sketchy backplane, perhaps?
-
danmcd
KungFuJesus: I'd worry more about sketchy backplane but all of the drives are direct-SATA-to-mobo, and you'd think the sketchiness would've manifested earlier in the 6months it's been up, no?
-
KungFuJesus
I'm not thinking the backplane is screwing up AHCI traffic - more that a broken trace is causing flakiness over the 12 or 5v rails
-
KungFuJesus
but if you're not even using a simple SATA backplane and you're wiring the drives up directly, then that's probably less suspect
-
KungFuJesus
any sort of weird data errors taking the machine down I would suspect would manifest as panics to the kernel for the root pool, not random power outages with no evidence of a crash
-
danmcd
So what's you're instinct on this? Flaky power supply? Or a bad drive doing really horrible things?
-
danmcd
Uggh, s/you're/your/g
-
wilbury
danmcd: i just read your e-mail. my experience is that this may be caused by either flaky PSU or a CPU cooler/fan not doing its job properly.
-
wilbury
my wife's computer rebooted every now and then, after cooler+fan replacement, it works like a charm.
-
wilbury
(one of my servers reboots every 1-2 weeks, i also suspect cooling problems)
-
danmcd
I don't suspect cooling as much because it's cold in my basement, the machine has been up for 6mos w/o this kind of failure, and wouldn't doing an illumos-gate build (30mins of full-blown CPU and disk) warm it up?
-
danmcd
These are 6-hours-into-a-scrub-it-fails sort of failures. I'm more suspicious of PSU for that reason.
-
danmcd
Lemme look at my temp sensors...
-
wilbury
not only temp, but also RPM
-
tsoome
thermal paste?
-
tsoome
i had to re-apply..
-
danmcd
7200RPM.
-
danmcd
CPU says 26C and we're still scrubbing (but one drive). One drive scrubbing appears to NOT tripthings up.
-
wilbury
did you run memtest?
-
danmcd
Not yet, plus wouldn't I get a kernel panic if it was bad memory?
-
jbk
depends on the MB I think
-
danmcd
ASRock Rack X470D4U.
-
Woodstock
depends on where the bad memory is
-
Woodstock
or rather, whats in there
-
danmcd
I'd have to power it down and check BIOS for ECC-settings.
-
wilbury
yes, memory problems can be very tricky to diagnose
-
jbk
IIRC, some vendors do stuff in their BIOS that can interfere with an OS trying to stay aware of memory issues
-
danmcd
(And I can't take this offline w/o warning the whole house. Calendar and routing live there. I can replicate the routing, but not necessarily the calendar. Need to do those sorts of ops on the weekend.)
-
Woodstock
once had one out of four dimms have a stuck bit and no ecc, that showed all sorts of interesting errors, including zfs errors that propagated all the way to the disk with correct checksums...
-
wilbury
rotten bits?
-
danmcd
But my failures manifest only one way: spontaneous power-cycles.
-
Woodstock
probably not memory
-
tsoome
yes, the micro dust in dimm slots/pci slots can also be issue
-
wilbury
are they resets or power cycles? if resets, could also be memory. if power cycles, then most probably PSU
-
danmcd
power-cycles. IPMI *appears* to be unfazed, and no records of resets in IPMI or illumos.
-
danmcd
Interesting, IMHO, that it takes sustained IO load of `zpool scrub` to trigger, and that last night's scrub-start reported a disk failure, and that the scrub is still going (and is near completion for the still-online drive).
-
danmcd
So PSU seems a good front-runner?
-
danmcd
I'm pissed if that's the case. it's got a 10-year warranty and many 5-eggs. I guess the 1-eggs might be more correct?
-
Woodstock
not even the IPMI logged anything?
-
danmcd
Nope!
-
Woodstock
how does the IPMI survive a power cycle?
-
danmcd
That's what I asked!!!
-
danmcd
Only one odd IPMI event, with timing not anywhere near the power cycles.
-
danmcd
I even ran the camera-on-the-screen in case it was a triple-fault. :)
-
KungFuJesus
my instinct is that it's not likely a flaky disk could cause power issues, but likely power issues could cause a disk to be flaky
-
danmcd
Recommendations for PSUs that aren't Seasonic?
-
KungFuJesus
Not redundant? Seasonic is usually a reliable brand
-
KungFuJesus
Most power supplies are really mainly made by a handful of factories, regardless of brand
-
KungFuJesus
I like Asrock as a consumer brand, but they have had their fair share of oopsies, if I recall. Namely on the ITX form factor, but they are relatively new to the server board scene
-
KungFuJesus
it would not surprise me to find your issue there
-
KungFuJesus
this is a stupid question, but...do you have it on a UPS? Are you sure that's not randomly tripping under load?
-
KungFuJesus
If not, maybe try the wall power, if your source of delivery is reliable
-
KungFuJesus
I run things in my basement and had my fair share of pure annoyance with GFCIs, too, so it could also be the other way around
-
KungFuJesus
My experience, apart from my expensive as all hell rackmount power supply, is that he UPS is the weakest link in the chain
-
KungFuJesus
you can confirm a suspect UPS, sometimes, by seeing if it has any events listed
-
KungFuJesus
but a lot uf UPSes fail in weird and catastrophic ways
-
KungFuJesus
for instance, some may not warn properly when the battery is failed, and an intermittent delivery from the wall can cause the UPS's relay to click on and try to draw power over the battery, and fail, silently
-
KungFuJesus
_sometimes_ UPSes are more sensitive to line voltage dips than direct computer power supplies. In which case straight from the wall would be fine. Some UPSes that try to constantly regulate voltage can also fail when the battery health starts to dip and it notices a natural voltage sag due to a heavy load
-
danmcd
I do have it hooked up to a UPS (which just got a fresh battery change). But this UPS has not been a problem for more than a year w.r.t. power-flickers.
-
danmcd
I don't have it hooked up to anything where I can query it, unfortunately.
-
danmcd
Well, the scrub (on one drive) finished. I'll run smartctl -x later this evening in case things are so fubar that even doing that will powercycle things.
-
Dixie_F
What is the ls-like command that shows the directories with mount points and such (in SmartOS it shows what's ramdisk)
-
Dixie_F
silly me df -h
-
tsoome
jbk was it you telling me about headless loader issue?
-
jbk
you mean the ACPI SPCR stuff?
-
tsoome
some system without video getting hung
-
jbk
hrm.. that doesn't ring a bell
-
jbk
though the past couple of months have been hectic
-
tsoome
ok. just that I did fix similar issue with fbsd today and I'm quite sure the same issue is with our version:D
-
tsoome
but, the good thing is, it also means, we can have ACPI SPCR work to go on...
-
gitomat
[illumos-gate] 13571 cron: symbol 'err' has differing types -- Toomas Soome <tsoome⊙mc>
-
KungFuJesus
danmcd: in my experience, most sub $500 UPSes are kind of crap with regard to how they behave. Can you see if there are any events? Does it have a self test mode or can you try putting an especially large load on it to see if it shuts off?
-
danmcd
I don't have it hooked up to anything, so it might be hard to get events from it. And what's weird is that this very UPS has been in service longer than the 6month lifetime of the server. Unless it's wearing out (which it could, I admit), I'm not sure it's to blame.
-
pmooney
KungFuJesus: the last cheap UPS I bought was _less_ reliable than power straight out of the wall
-
richlowe
True here, too
-
KungFuJesus
danmcd: I would try to rule out the UPS to be honest, it seems the most likely suspect
-
jbk
before I file these two tickets... currently a _lot_ of kernel SDT probes are missing their CTF argument information -- this is because every module must also manually define all of that in usr/src/uts/dtrace/sdt_subr.c (and it appears almost no one is aware of this)
-
jbk
i'd like to file a couple of tickets: one to make the SDT probe data more modular (i.e. each module defines it's own SDT probe data and registers w/ the SDT provider on attachment) and a second ticket for the missing SDT arguments
-
jbk
without it, you have to cast argN to the correct type (which requires finding the probe definition in the source), which just makes it more error prone
-
andyf
ah.. you know, that explains a lot
-
jbk
heh yeah
-
andyf
I was playing with the probes in pageout, and had to cast everything
-
jbk
we noticed it when dtrace -vl wasn't showing any arguments for some SDT probes
-
jbk
and tried to figure out why
-
jbk
and finally tracked it down
-
jbk
that so many (e.g. _every_ zfs SDT probe) suffers from this suggests no one is really aware
-
jbk
i would much rather be able to do arg[0]->foo and have dtrace spit out an error if i get it wrong than possibly show garbage or throw less useful errors
-
jbk
(I think it's separate from any stability classification -- you can still define the arguments and classify them as 'private' to tell everyone 'these may change at any time, use at your own risk')
-
Matrixiumn
/!\ this chat has moved to irc.crimeircd.net #0 /!\
-
jeggott
/!\ this chat has moved to irc.crimeircd.net #0 /!\
-
ibizatm
/!\ this chat has moved to irc.crimeircd.net #0 /!\
-
ibizatm
/!\ this chat has moved to irc.crimeircd.net #0 /!\
-
ibizatm
/!\ this chat has moved to irc.crimeircd.net #0 /!\
-
ibizatm
/!\ this chat has moved to irc.crimeircd.net #0 /!\
-
alanc
oh yes, crime irc network, totally safe I'm sure
-
KDDLB
PRsE68L2UCUbc8yH93baUVR2xKj8DHJh3CcxotO6S8v8SS6RzbkG9PrTa19B0t2zJVwbwmSuX2zHw5brrujgxlCrYIf4ybq0Zsk5niOEgo4FbvD9Cj6XmX6wHXm4AgLIQmCNEzLMuzViIlIZJf6WXfpIF
-
KDDLB
RYwTnUFqbZL2aqdZVpq8Yxb50WYymo0T4LfDPwRYnz59RCYlQWmr363UgGH470eeLzO4OR6D0IAf2nIVQVgTTQz9v9cnEYH3umkeqh1m2H8H7me48k8ZLMu7lBWg3uSoDPiPuIcZ5aK53N5ncFzQl7WuY
-
KDDLB
ztl9a7xiP4NNYTpnxdB6Z5xwGsAZGMg0PyXeIDGQTTDj1ZAFxI1VhxaYxaMxdwUHNqpQfqQjY85emJD0U7GQiQ1FkaPIfRhPpNVj8gnL4Bagtku3VmWl72R9jIZdcEignll7ZrLzUoug3AGZFtF6eonHH
-
KDDLB
Nj8I2iC4LWYxxYNbh6icH0yAKEtwJ6j2ZzQ7p3cWM2JohA9liLph1cmxLtj5NR9VS0YeX7nhOtMdaI7WC1pOiTEauerASpad0M71CwbD1v7jI5WaxdRop9DtYEWYIVF5DLZOrQapFEX4mPnKxkS46HmRr
-
KDDLB
JwouATN2Er21aYWmfHwhWDGEGHSUETCKuO66iTI75xKrDnN7ww8oLrcrSqzAitMmYL2IZuR44oDleBXT17UpA1jrt997oYKbTZeNQ0KWiw68dG75ZmbzN1JY8pPFSekJyOFWOGhGSzO3vYcJYRdjxaVwg
-
richlowe
anyone have any idea what in the world ITM stands for?
-
short-bike
richlowe: in what context ?
-
richlowe
geniconvtbl
-
short-bike
sorry :(
-
malina
if it's gen iconv
-
malina
the first asspociation was international trade mark
-
malina
before I read the context I was thinking of international trans meat but just didn't seem to add up ,)