Rendered at 16:17:11 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
a10c 1 days ago [-]
My action failed with "Unexpected error fetching GitHub release for tag refs/heads/master: HttpError: Sorry. Your account was suspended"
Which certainly made me shit myself, briefly.
neya 1 days ago [-]
It's an eye opener. Think about it - today, it was a mistake. But, what if it really happened? What if you really lost access to all your years of hard work? It's a wake up call. A blessing in disguise to store what matters to you the most locally, backed up offline. Never trust any single provider. Be it MS or Google or Apple. RAID is the way.
onion2k 1 days ago [-]
People should use something that keeps a local copy of their code and just copies it to Github and to other contributors with a sync process to push and pull changes. Some sort of 'distributed source control system' maybe. Then people would only need a 'hub' to connect to people, and it'd be easier to move somewhere else.
gopalv 24 hours ago [-]
> Some sort of 'distributed source control system' maybe
The day it broke away and became centralized was when we had a PR + mandatory "Required actions" to merge to main.
ruszki 22 hours ago [-]
That’s only mandatory on the “hub”. I can do that locally anytime.
bergie 17 hours ago [-]
I'm looking at setting up rngit mirrors of all my repos on our boat NAS. Conceivably it also allows issue tracking and collaboration without centralized infra
What you just described is Fossil. It has an auto-sync feature that makes everything feel distributed.
Just set up a Kubernetes deployment and you’re set.
But as others mention, GitHub’s primary strength is collaboration. If you want decentralized, solve this by creating a decentralized collaboration tool on top of fossil and/or git.
For example, how to do pull requests and code reviews?
40four 24 hours ago [-]
Why they just described is Git :) pretty sure it was a joke
marricks 24 hours ago [-]
I like how tech seems to be all about stacking more and more turtles on top of each other:
Gosh, it's hard figuring out what changes Lorne made if only we had a system to merge those changes. Enter git
Gosh it's hard figuring out what packages Rachel had to make this work. Enter rubygems/pip/npm
Gosh it's hard figuring out sync these changes across a network. Enter github
Gosh it's hard figuring out how to get those packages working on my operating system. Enter docker
Gosh centralizing our distributed version control software system onto one website is getting really unreliable. Enter fossil(?????)
If we go any further having one computer per business with a sign up sheep is starting to sound pretty fucking attractive.
coldpie 1 days ago [-]
This gets tiresome. Github is a lot more than a host for Git repositories. If you want to suggest that people use something else, you need to suggest a replacement that has the features people use Github for.
ornornor 1 days ago [-]
Increasingly less and less so as they “upgrade” their offering and have more and more downtime.
danudey 24 hours ago [-]
I think you missed the joke, which is that the parent poster you're replying to is suggesting a 'solution' to the problem which evolved in complexity until he was just describing Github again.
doctorpangloss 1 days ago [-]
yeah, #1, it is free private file storage, and #2, it's a download portal for free as in beer software replacing paid offerings. that's what it is for 99.99% of people.
being a host for git repositories has never been its core competency. neither has its groupware offering.
does it even serve OSS well? a very interesting criteria is, "Have mature or adopted end-user-facing OSS recently merged a large PR from an unallied contributor?" The answer is overwhelming no. This is why there is so much innovation in this space.
mpaco 1 days ago [-]
I recently got my GitHub account suspended for 4 months. When it was finally reinstated, their support just said it was a "mistake".
Proudly self-hosting Forgejo since then.
MatthiasPortzel 24 hours ago [-]
This happened to me as well—thankfully not my personal account that I use for work, but the organization associated with an open source project I worked on was suspended. It similarly took 2 months for GitHub to restore the organization.
> Our team is currently experiencing an unexpectedly high volume of tickets which has resulted in longer response times than we prefer. We acknowledge the long wait and apologize for the experience.
> Sometimes our abuse detecting systems highlight accounts that need to be manually reviewed. We've cleared the restrictions from your account…
Fully self-hosted IMO can be an overcorrection. The issue isn’t “relying on other people”—it’s relying on GitHub, when they’ve made it clear they don’t care about uptime and they don’t care about support turn-around-time.
SpaceNoodled 11 hours ago [-]
I care about uptime and have instant support turnaround. Self-hosting sounds like a great solution.
corvad 1 days ago [-]
RAID is not a backup.
PokemonNoGo 1 days ago [-]
They... Didn't describe RAID? More 3-2-1.
filleduchaos 1 days ago [-]
The last sentence in the comment is literally "RAID is the way".
jrockway 24 hours ago [-]
I think they were intending to evoke the image of RAID rather than literally referring to a redundant array of inexpensive disks. You host your code on Github, Gitlab, and at home, then you survive a Github outage. It's a redundant array. Not sure it's inexpensive, though.
iso1631 1 days ago [-]
Well yes, my git repositories sit on my laptop, that's the entire point. If github banned my country because its president has a tis, I can push my entire commit history to another company. Same with anyone else who's working on it.
It would be a pain as I'd have to set up a few integrations again, but github is far lower down the risk scale than the vast majority of SAAS providers
bulbar 11 hours ago [-]
They rely on GitHub actions, not the repository itself.
I hope people here are aware that you can push your repo somewhere else if wanted.
Git is a distributed system, there isn't even a server, only other git repo instances that are remote.
iso1631 6 hours ago [-]
I rely on actions, but those actions are pretty much "on this type of change to this branch run these scripts"
It will be a hassle to migrate to another platform, possibly a couple of hours work to do the 25 repos in my ~/git/ directory.
Even highly complicated actions can be migrated quite easily -- the source is stored in .github/workflows/blah.yml
Same. It's weird how I always find out that GitHub is down before GitHub does. Took 15 minutes before it appeared on githubstatus.com
jaapz 1 days ago [-]
All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage.
logifail 1 days ago [-]
> All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage.
Is it true that official service status pages are updated automatically?
baby_souffle 1 days ago [-]
> it true that official service status pages are updated automatically?
Depends. Typically no because there’s an art to crafting the actual message around impact… but sometimes yes it is automated
logifail 3 hours ago [-]
> Typically no because there’s an art to crafting the actual message around impact
I was thinking more of needing to notify/get sign-off from management...
hnlmorg 1 days ago [-]
You'd expect them to be monitoring more than just the HTTP response codes from user requests for precisely this reason.
If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well.
But effective monitoring is harder than people assume.
dncornholio 1 days ago [-]
> If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well.
Isn't that what monitoring actually is? The issue seems to be in their testing, not monitoring.
hnlmorg 1 days ago [-]
No, monitoring for HTTP response code is a subset of observability and not one that generally gives you the best insights into which subsystems are misbehaving nor why.
There are synthetic tests, where you can generate API request calls or even simulate an entire user journey. These allow you to control the user agent, the payloads, and thus you know anything errors back are actual errors. These are triggered by the observability platform (think like running a cron-job) and thus you're not tied to user activity to see when problems arise.
There are other metrics outside of HTTP response codes too. Think like free RAM, CPU usage, disk space, etc. This is just naming some obvious ones because these types of metrics are generally bespoke to the type of application your monitoring. And with these types of monitors, you'd not just have an alert when things have failed, but ideally have alerts when an irregular trend is showing that things are likely to fail too. This latter type of monitors helps you get ahead of the problem before it become customer facing.
Then you have more traditional stuff like logs. This will also be bespoke to the application. But you'd expect errors in logs to get surfaced quickly. Assuming Github have good hygiene in what's being logged.
Tie that up with APMs, RUM, and other goodies like that and you'll have diagnostics to investigate issues when they appear.
(this is just a super high level view of observability too)
lokar 1 days ago [-]
Even a synthetic probe needs a few failures to trigger an alert.
You should not alert on cpu, ram, etc
hnlmorg 1 days ago [-]
> Even a synthetic probe needs a few failures to trigger an alert.
It doesn't "need" that. That just how most people set it up because it’s an easy sane default that allows for network jitter without inexperienced engineers thinking about different conditions triggering different types of responses.
If you’re measuring internal APIs from an observablity solution that’s has nodes already inside you’re network enclave, then there is a strong argument for alerting early.
> You should not alert on cpu, ram, etc
That’s not true to say as an absolute statement. And a generalisation it heavily depends on the system your monitoring and how it behaves under pressure.
But in any case, I wasn’t suggesting CPU alerts were the end goal. I said:
> these types of metrics are generally bespoke to the type of application your monitoring.
Ie you’ll use metrics but those metrics will be highly specific.
The CPU examples were an illustration as to what a “metric” is (it might seem obvious but not everyone is an expert) but the point was HTTP response codes aren't the only types of metrics one should be capturing and watching.
lokar 1 days ago [-]
Ah, yes, I misunderstood. And I have seen cases where a direct CPU alert makes sense, but 99 times out of 100 times I see it, it's nothing but trouble. Worse, I tend to see the cpu alert when there are no end to end synthetic alerts, 500 alerts, queue depth alerts, etc.
If your requests are fast and cheap, you can probe frequently relative to your goals, but often that's not really possible (think, long SQL queries, or scheduling a container/pod). There you need several datapoints, or possible fewer augmented with other signals.
hnlmorg 1 days ago [-]
Yeah very true.
Talking about long SQL queries, I quite like throwing CPU alerts on database servers. They'll be a low priority alert (ie no out of hours "pagers") so just something that goes into a slack channel. But they're a good indicator of when developers have poorly optimized SQL, or the DB schema is poorly defined (eg missing indexes), or the DB server itself is poorly sized.
This wouldn't be something you'd expect to need in production and definitely not something you'd rely on as a notice of a production outage. But it is an example of one of those 1% occasions where a CPU alert does add value to the overall observability of the application.
But this also ties into your excellent point about how you'd use CPU and other data points to build a picture of what's happening in your application.
lokar 24 hours ago [-]
Oh, I was thinking about it as the person running SQL as a service. People run queries that go on for days....
idle CPU is often wasted CPU
re-thc 23 hours ago [-]
> But effective monitoring is harder than people assume.
Who says public status page equals internal monitoring.
They likely know faster than you. Whether they post it publicly is a different issue (hint: SLA penalties, news impacting stock etc)
hnlmorg 23 hours ago [-]
I never mentioned anything about status pages.
Are you sure you’re replying to the right comment?
re-thc 22 hours ago [-]
> I never mentioned anything about status pages.
For context, the parent comment you replied to started with status page.
Then are you talking about internal leaks or just guessing? Otherwise besides what's public how do you know they don't know?
hnlmorg 21 hours ago [-]
It was two comments prior to mine that mentioned status pages.
Someone then replied about how it takes a bunch of HTTP response errors for problems to be alerted and thus I commented that application observability would consist of more than just waiting for users to hit errors.
echelon 1 days ago [-]
In a high performance service with good maintenance and upkeep, you page for all 500s. A noisy pager forces the team to fix the 500s.
Maybe the Github Actions infrastructure isn't run like that.
Which makes me think a small amount of random issues which happen even though nothing is broken, is normal everywhere. Especially once move things around on a network, there's potential for a lot more random errors.
bobthepanda 1 days ago [-]
It’s where monitoring for 9s is more important at that scale than absolute errors. So long as degradation is graceful or retried it should not be a massive problem.
It does require constant tuning and adjustment though.
KPGv2 1 days ago [-]
Bitflips are something that can happen in consumer-grade RAM, so that tracks (and it's comforting that wayward cosmic rays are a substantial reason for an application's crashes!), but on enterprise servers, they will run ECC RAM that is very resistant to bit flips.
This is why data hoarders who have NASes with lots of space insist on running their servers with ECC RAM despite it being significantly more expensive. Because bit flips, for all intents and purposes, cannot happen. The RAM itself detects and corrects for them.
I wouldn't expect bit flips to be a significant contributor to enterprise problems.
Anon1096 1 days ago [-]
Bitflips specifically may not be; things like network issues, noisy neighbors, row/rack/host maintenance (leading to a downed and migrated host) absolutely are things that happen at high frequency at scale and cause your background level of errors to be more than 0.
maccard 1 days ago [-]
You've completely missed the point - It's not about bitflips it's about errors that are outside the scope of what's fixable.
KPGv2 21 hours ago [-]
I suppose I misunderstood what the "random error" was supposed to mean. I wouldn't call a network error a "random error" because it's caused by things that are internal to the system (entities using a network). A bit flip is caused by an external factor: cosmic radiation. To me, that's what a "random" error is.
If your network goes down because of a DDOS, or part of your system overheating, that's an internal issue you had control over.
If a bit flips because of cosmic radiation, you can't really do anything about that, and it's utterly unpredictable. That's "random" to me.
TheDong 1 days ago [-]
Do you know of a single service at a single company that actually does that?
I know all of Gmail, every GCE service I can think of, every AWS service I can think of, Amazon.com, Netflix, and Github all do not page on just a single 500.
I know none of those are particularly "high performance" though. Curious where your experience is coming from.
CBLT 1 days ago [-]
I've been oncall for a different G service that nearly paged on every error. It used the standard error budget tooling, but on hundreds of user buckets because the engineering around locality-specific configuration was... suspect. Many of these buckets had single-digits user. If a user was on their phone and lost signal, I was paged. Very poor oncall experience.
theta_d 1 days ago [-]
The sub-service at IBM cloud I worked on had an insanely small error budget such that pages were nearly constant. On call was hell week until a few of us insisted on fixing the issues. The "few" of us were contractors. The employees seemed more than willing to just let the pages continue.
alexfoo 24 hours ago [-]
Some companies pay more if people are paged. It can create a perverse incentive not to fix problems or, in extreme cases, to watch things going wrong, waiting for the page, and then being ready to fix it straight away.
echelon 1 days ago [-]
I worked at a large fintech moving billions of dollars in volume a day.
I had a fairly long tenure, where I maintained multiple key services in critical online payments flow. Authentication, authorization, core business and risk data, as well as some cross-cutting control plane stuff, etc. You needed one or more of our services to take a payment, serve any request from the employee dashboard - pretty much everything hit our services. The entire company ground to a halt without my team.
We paged for every single 500. In instances where a particular class of 500 was spurious or not worth fixing, we would leave it acked or mark it as noise. But typically we'd just put in a fix as soon as possible so we didn't page.
Our graceful shutdown and traffic shaping stack was great, but occasionally we'd get a few pages during deploys or failovers.
Oncall was typically not bad, but when it did get bad it was terrible. I've been involved in huge outages that cost hundreds of millions of dollars. Usually it was the fault of multiple teams having compounding runaway failures rather than one service or bug in particular.
It's inexcusable to have a customer's payments not go through. We engineered around resilience. We had strict five nines SLAs and p99 targets and evaluated our adherence with even the smallest partial outage. Hundreds of other services depended on ours, and downstream impacts were huge, so we had to keep a tight ship.
We didn't have "business hours"-only paging either as our platform was available globally, including a heavy install base in Asia.
sunrunner 1 days ago [-]
> We paged for every single 500.
Assuming the existence of some kind of network (with zero guarantee of 100% reliability), how does this work in practice? Is each 500 treated as an event that needs investigation, even if the result of that would end up as 'a router dropped something from an internal buffer but the transaction as a whole was re-tried by a parent so the service itself recovered'?
LPisGood 1 days ago [-]
A reliability engineer from Jane Street gave a great talk about this, five nine’s of correctness in reporting, etc isn’t enough for the SEC.
Client network timeout shouldn't result in 500. With 408 and retry you should, dependent on the business criteria, get either an upsert (transaction is retried) or 422 (validation that given entry already exists).
Even if it's "DB in datacenter I tried to save to was hit by meteor" event, you can cater for this not to result in 500 (ie - DB unreachable, retry in a couple of minutes); the question is if you want to.
compumike 1 days ago [-]
Re: "page for all 500s": there's a world of difference between "page me with a critical alert at 3am" and "notify me on Monday morning when my normal workday starts". At the extremes:
If my DB health check endpoint is returning 500s for N consecutive checks over M minutes, yeah, please wake me up at 3am!
If one user hit a weird edge case in form validation and got a one-off 500, please don't! We can fix that on Monday.
Not always easy to distinguish those clearly or configure those business hours rules, but for my team at https://heyoncall.com/ that is the goal -- otherwise your team burns out fast. Waking up someone at 3am has a real cost, so you better be sure it's worth it.
wasmitnetzen 1 days ago [-]
Shouldn't Github be large enough to not have anyone on-call, but just rotate the responsible team around the world?
alexfoo 24 hours ago [-]
One team can't troubleshoot AND FIX every possible subsystem, so you just end up with lots (growing to hundreds) of people "on-call" anyway.
As others have said, follow-the-sun type models do exist, usually staffed by people in their normal working hours (EMEA, Americas, APAC) but this means you've still got to cover the weekend and public holidays (which there are a lot of when you factor in plenty of different countries).
Where you need a quick response you can have a core ops/noc team that looks at things with lower thresholds and shorter windows, and their job is to do the initial triage and then page the appropriate team earlier than they would have been alerted by their own alert thresholds/monitoring.
Actually clicking the button to change the status on a public status page is a whole different topic that becomes very political in certain companies.
bobthepanda 1 days ago [-]
At least when I worked at a Bigcorp a lot of that was being cut to save costs.
lokar 1 days ago [-]
I've worked in large orgs where we could (at at some times did) have around the world rotations. They don't work well. It've very hard to maintain real team cohesion, and you end up with really superficial operations. People tend not to dig in really deep, find good fixes, etc. Lots of superficial bandages.
awithrow 1 days ago [-]
that is absolutely not the case for any system of size and scale. that would just burn out the on-call team and not result in improvements. Error rates/budgets are used instead.
hnlmorg 1 days ago [-]
It depends what you're monitoring. If it's response codes from user generated queries, then I'd agree with you.
But if it is synthetic queries sent from the monitoring platform, then you control the user agent, payload, and endpoints. So any failed requests are a symptom of a misconfiguration and/or failure that should be investigated. Albeit not necessarily as a P1 priority.
hvb2 1 days ago [-]
> A noisy pager forces the team to fix the 500s.
I'm sure you're not in ops. Or in a dev org of a service with decent request rates.
What you're asking for is a service to fail silently. There's no way a service with a decent request rate to have 0 500s. Not when it still sees development.
A 50 year old bank API? Maybe...
rhyperior 1 days ago [-]
You only do this when you’re trying to use incident management as a hammer to make a point to somebody whom you have otherwise failed to convince to fix something through persuasive argument. Ie, it’s punitive.
swiftcoder 1 days ago [-]
Yeah, no, nobody runs cloud services like that. At AWS most alarms required failures in 3 consecutive 5 minute periods. Critical things could be on 3 consecutive 1 minute windows - but that alarm starts a 15 minute escalation for the oncall engineer to check in, and they have to validate the issue isn't a false alarm before updating the status page would even be considered
jordemort 1 days ago [-]
forget it, Jake; it’s Azure
registeredcorn 1 days ago [-]
I'm not arguing with what you're saying, but it does make me wonder: What exactly is the point of the status page, if "it is normal for users to already see errors before GitHub officially counts it as an outage"?
Is it more so to have something to link to for managers who aren't using the service have a pretty bar to look at and feel like they are "doing something"? Or is it more of a kind of a way to prevent confirming what you already suspect to be true. E.g. "Huh. Me and Jim are seeing problems. How about you Tom? Oh wait, crud. The service page is confirming it's down now. Never mind! Who wants coffee?!"
filleduchaos 1 days ago [-]
There is oddly enough a middle ground between "zero errors whatsoever" and "outage".
simonjgreen 1 days ago [-]
More likely that 'update the Status site' lives a long way down their incident response plan, and they have alarms going off well before that
jordemort 1 days ago [-]
yeah I mean a company the size of GitHub certainly can’t be expected to have enough staff to walk and chew gum at the same time
swiftcoder 1 days ago [-]
If it's like other BigTechs I have worked at, you need director-level signoff and comms team approval to post an outage notice
PunchyHamster 1 days ago [-]
it should be automatic tho. Probably isn't so they can at least get the one nine on availability
simonjgreen 1 days ago [-]
Marketing definitely takes interest in status sites
re-thc 1 days ago [-]
> It's weird how I always find out that GitHub is down before GitHub does
No, it's not. Official updates = potential SLA penalties. Always requires approval.
drcongo 1 days ago [-]
This is the most plausible reply.
24 hours ago [-]
chrisjj 24 hours ago [-]
> githubstatus.com
There's a threshold. It shows only once 1000 users complain.
/i
1 days ago [-]
ridiculous_leke 21 hours ago [-]
> Which certainly made me shit myself, briefly.
Can you sue companies for inducing such anxiety?
Imustaskforhelp 21 hours ago [-]
IANAL, but I can probably imagine a case being made if a person really got so stressed that for example any health condition got invoked from the stress. It might be up to the lawyer to explain how exactly the service caused the stress and its direct relation to health condition though and up to the judge.
but I suppose that there might be some terms of conditions within using github (ahem Microsoft) that you can probably not sue them for something like this.
It really depends upon the severity of situation (imo)
For example, if a person had any heart condition and they got so stressed because of an error at github (which to be fair, I can understand the stress part, imagine losing some part of your software because it was on github and the amount of direct damage to livelihood if your income depended on it)
and I think that the judge might have to be in just the right technical know-spot as well and someone who can understand the situation from programmer's perspective hopefully.
Then I can see a case being made.
once again not a lawyer but an interesting question, would love reading other replies to your comment.
also for what its worth, you can sue any company for X,Y or Z. The question worth asking is if you can win such lawsuit.
Personally I believe it might be hard but not impossible but for all practical use cases it might as well be but the only answer can probably be found in court. I am just guessing at this point.
dvduval 1 days ago [-]
Yes, Thais can be be really frustrating when you’re trying to get work done. There needs to be more competition and better alternatives and the LLMs need to offer easier connection to these alternatives.
weird-eye-issue 1 days ago [-]
What do the Thai people have to do with this? :(
denisw 1 days ago [-]
Pretty sure that they wanted to write "this", typed something different by accident, and auto-correct struck.
weird-eye-issue 1 days ago [-]
Oh gee thanks
superxpro12 1 days ago [-]
Reminded me of the "Thai Fighter" joke from family guy's star wars spoof lol
1 days ago [-]
cpfohl 1 days ago [-]
Wasn’t my fault this time! I haven’t started work yet.
Hah, I know the feeling. I installed Ubuntu on a PC recently, it obviously happened to be one of the days they got DDOSed and apt repos were unreachable. I had other things to take care of, so I put it aside for the next week or so. It didn't help very much, cause after picking it back up, halfway through, Snapcraft went down.
Waterluvian 1 days ago [-]
Yeah but you thought about it, didn’t you?
cpfohl 1 days ago [-]
I did....maybe my powers are growing.
JsonDemWitOster 1 days ago [-]
Sorry guys it might be me.
I vibe coded a script that interacts with both Gitlab and Github via their APIs and I've been using it pretty heavily since this morning. I crossed the streams! Goodness, I didn't know it would be _this_ bad!
zombot 1 days ago [-]
It's only natural that this kind of promiscuity provoked an allergic reaction from Microslop.
thesdev 1 days ago [-]
Next thing you're gonna tell us you're SRE at GitHub.
swyx 1 days ago [-]
> I haven’t started work yet.
spooky action at a distance
Andrex 1 days ago [-]
Uh oh. That means there's at least one more like you out there that we don't know about.
cpfohl 1 days ago [-]
I always wanted superpowers, but I never dreamed it'd be like this.
- So many super-heroes/super-villains
ramon156 1 days ago [-]
Was about to send my bill to you.
... You're off the hook this time./s
keepamovin 1 days ago [-]
[flagged]
bouk 1 days ago [-]
Insane, we have to come up with contingency plans now for long-duration GitHub outages because we can't safely do deployments. For a service we're paying thousands of $ per year for even though we host runners ourselves...
decodebytes 1 days ago [-]
Same thoughts - we use an action to ship to production, its builds an image, pushes it to ECS which triggers a deployment.
We can't be blocked here. Seems silly what we settled on this, but for a long time GitHub had been reliable enough for many years, but things are sliding down the pan as of late.
mystifyingpoi 1 days ago [-]
Sounds like a very easy process to rewrite in bash/python and have it on hand if needed.
Salgat 1 days ago [-]
It's funny, when we were acquired they started moving us to Github actions but it seems that maybe we should stay on our old crusty self-hosted Jenkins setup...
cryo32 1 days ago [-]
You should never entirely depend on a third party service for deployments.
Been burned too many times on that one.
999900000999 1 days ago [-]
Ok.
Move to EC2.
Darn AWS is down.
Alright, run it on a Mac Mini in your basement. Ahh dawn, your ISP is having issues. Good thing you have a backup 5G hotspot.
Ohh no, the power is out.
Eventually you have to trust someone else.
GitHub is a tragedy of the Commons. Too many people are using it, and Microsoft isn't willing to handle it correctly.
Feels like a very good business opportunity. Minimum 50k yearly contracts, GitHub with actual uptime. GitPro ?
cryo32 1 days ago [-]
We’re actually moving back to redundant data centres due to all of those problems.
Aggregate risk is too high.
sleight42 1 days ago [-]
It's almost as though GitHub should never have let itself be sold to Microsoft...
999900000999 1 days ago [-]
I'm sure the VCs who invested in GitHub disagree.
This is supposed to be Hacker News! Who is coming up with a startup to fill the gap !
bee_rider 1 days ago [-]
Maybe we need a split between source management and distribution? The former looks like git[hub] to me, the latter maybe more like a Linux distro repo?
bouk 1 days ago [-]
We could still deploy manually but it's suboptimal! And we're 'flying blind' without CI runs
matt_kantor 1 days ago [-]
> And we're 'flying blind' without CI runs
You should never entirely depend on a third party service to run your tests, either.
cryo32 18 hours ago [-]
make test
Should work without CI
dnnddidiej 1 days ago [-]
It is a control pain
the8472 1 days ago [-]
./deploy.sh
yoyohello13 1 days ago [-]
Self host gitlab. If you already host runners it’s not a big lift.
xaerise 19 hours ago [-]
Even if there is features that are similar, most of gitlabs features are for paying customers only.
yoyohello13 16 hours ago [-]
OP said they already pay for GitHub. We pay for the premium tier of Gitlab at my work and it’s definitely worth it.
Cthulhu_ 1 days ago [-]
It's always best to be portable - always be able to do builds and releases locally (at least, once you get the keys - it shouldn't be possible by default), then add things like github actions on top as convenience.
sebmellen 1 days ago [-]
Same here. You’d think they could at least separate out the GitHub-hosted and self-hosted runners, so you’re still able to dispatch jobs if the self-hosted runners are down.
ketzu 1 days ago [-]
If the job queue is down, that wouldn't help, would it?
On my repo the jobs do not get scheduled on the PRs at all, so I assume that separation wouldn't help for todays issue.
I’m not convinced they actually do, because GHE on the cloud tends to have the same problems as the main outages. Probably costs extra to be “single tenant” or whatever
sofixa 1 days ago [-]
Depending on how many thousands of $ per year, it would probably be cheaper and more reliable to self-host GitLab. It's better in terms of organisational structure (you can have one, including access and secret inheritance), and (personal view) Gitlab-CI is better than GitHub Actions because it doesn't push you towards a JavaScript/NPM style dependency hell. And it's actually fairly easy to self-hosted, with options from a single machine with an omnibus package that handles everything to a full blown autoscaling Kubernetes deployment.
hsbauauvhabzb 1 days ago [-]
Sounds good until you see their cvedetails page
lazystone 1 days ago [-]
Hide it behind VPN, so it's not accessible from outside.
hsbauauvhabzb 17 hours ago [-]
Now patching becomes a responsibility, unless your organisation is willing to run knowingly vulnerable software.
PunchyHamster 1 days ago [-]
When you own it you can just limit it into vpn-ed company users, that significantly cuts down on the area that can be hit
sofixa 1 days ago [-]
I mean, the GitHub Actions supply chain risks and attacks definitely compensate for any GitLab security vulnerabilities you can think of.
user43928 1 days ago [-]
[dead]
re-thc 1 days ago [-]
> For a service we're paying thousands of $ per year for even though we host runners ourselves...
Wait until you charge you for self-hosting runners.
Oh wait. They already tried.
pluc 1 days ago [-]
Sure. Don't use GitHub.
You can now hire me as an overpriced consultant instead of paying Microsoft.
bob1029 1 days ago [-]
The last two projects I built I did the CI/CD manually with a small win32 service that polls git and builds+deploys the main service locally. It's barely 200 lines of code. Not much to go wrong. "dotnet publish" is not difficult to wrap.
The latest language models have enabled this sort of thing for me. I can integrate a mini Jenkins into every project within a 5-10 minute prompting session. This sort of code isn't hard. It's just tedious, and the LLMs absolutely rock at boring repetitive stuff. Having a win32 service start up successfully on the very first try is something I haven't experienced until 2026.
starik36 1 days ago [-]
That works for relatively simple scenarios. When you have to add deploying sql changes or something having to update something in the cloud, you'd have to include a lot more plumbing.
Yokohiii 1 days ago [-]
In my world CI/CD and db migrations are 2 different things working together. CI/CD at heart is rather simple for many setups. Migrations need quite a lot scrutiny, you really want to mess up there. But if you run on gihub actions with 50/50 uptime, does it matter?
bob1029 23 hours ago [-]
Deploying SQL changes is actually trivial if you are using SQLite.
I agree in a hosted+shared SQL scenario you have to be a little bit more careful with all of this. Arguably, you should have a separate schema management phase in these cases.
But if you are just SQLite embedded in the service, you can use the user_version pragma to track schema version and perform deterministic migrations (assuming a user didn't manually jack with the file in-between).
peheje 1 days ago [-]
Deploying SQL changes? Why not just let the application do that on startup.
Ofcourse be backward and forward compatible. SQL change only deploy.
"Update something in the cloud" <- What do you mean?
Yokohiii 1 days ago [-]
> Why not just let the application do that on startup.
That only works on extremely simple setups and has risks. If you have only a single server, you can stall it. Now, how to roll back?
peheje 21 hours ago [-]
We try to keep things simple. Everything has risks. No stall, run async, backward compatible. DB handles rollback via transactions. Happy to expand if interested.
k4rnaj1k 1 days ago [-]
[dead]
ValentineC 1 days ago [-]
Not reflected on GitHub Status: most of the frontier models disappearing from most people's subscriptions:
As an Indy hacker I want to see GitHub succeed, but I ditched actions years ago - (shocking) false economy. Spend entire nights pushing to actions over and over only for complaints about weird/niche dependency issues and other oddities - the cycle time's just too slow and the DX is no fun (my pain doesn't even factor in outages; just the feature itself as it's intended to be experienced). I want to spend time talking to users and building features, not debugging weird syntax or dependency issues on a remote machine non-interactively.
So why are Actions so unreliable anyway? Occam's Razor would probably suggest the domain is inherently complex/difficult; but other providers show that reliability is possible. What would Occam's Razor suggest next? Poor management..?
24 hours ago [-]
frisbee6152 1 days ago [-]
What did you switch to, and what do you like about it?
nomilk 1 days ago [-]
Running tests locally. It's primitive, but incredibly reliable, and a breeze to debug if (big if) there is any dependency issue.
gchamonlive 22 hours ago [-]
I have Gitlab with a runner on a notebook I have running as a server. Pretty solid and if you need to bail on Gitlab SaaS you can BYOI and selfhost. Plus the CI is many streets ahead of GitHub in terms of pretty much everything.
xixixao 24 hours ago [-]
How do you ensure you or your contributors didn’t forget to run the tests?
You’d need at least some hash of sources + test results, and check that it matches that (in CI).
And you’d still deal with environment differences.
nomilk 23 hours ago [-]
> How do you ensure you didn’t forget to run the tests?
Reasonable concern. In ~10 years of indy development, I haven't forgotten to run tests before pushing to main, ever. So setting up and maintaining complicated machinery to solve a problem that could (but never has) happened doesn't justify taking focus off other more important things, namely building.
The benefit probably increases with team size (I'm a team of 1, so I appreciate the luxury of being able to dodge CI/CD entirely).
csomar 23 hours ago [-]
So you switched to nothing? That’s not the purpose of github actions or remote ci/cd. Anyone can run tests/builds locally.
nomilk 22 hours ago [-]
I think it comes down to risk tolerance. For an established company that wants to avoid upsetting users at all costs, CI/CD makes sense. But for a nimble 'move fast and break things' startup, it can steal dev time for very little upside.
Say a disaster happens and someone pushes to main without running tests, 9 times out of 10 it will be of ~zero consequence (either the code works first time, it was a cosmetic change that hardly affected users etc).
I know there are horror stories and CI/CD would have prevented some of those, but IME they're just not that common nor severe for small operations, and even when they happen, only a small subset are irreversible/unfixable.
csomar 21 hours ago [-]
That's not the purpose of a remote CI/CD. Your pipeline can be as strict or as loose as you wish. It's there to show you a log of the execution as it happened in a neutral environment (remote server).
Basically, what you are suggesting is that everyone advertises their tests/builds run on slack? Also when two devs merge their changes, who compile/tests the master branch?
nomilk 21 hours ago [-]
I see the benefit (it avoids the “works on my machine” problem), but my rails app isn’t too fancy and works on heroku ~100% of the time when it works on the dev machine. Making an intermediate build redundant (technically not entirely but it’s just not worth the effort).
For small teams it could be as simple as everyone agreeing to ensure tests pass on main before pushing to prod.
juanre 23 hours ago [-]
A good Makefile goes a long way.
peterspath 1 days ago [-]
I moved a while back to Forgejo -> https://forgejo.org couldn't be happier. Highly recommended.
Looks lik a terrible source. Like someone ran Claude on the codebase, didn't analyse the results, then vibe coded a blog post. And the dustri.org link doesn't work for me
Incredible how reliable the heuristic of "something seems off - probably github being down" has gotten these days
comboy 1 days ago [-]
It's big enough that every time it goes down, it surely stops somebody from pushing fix for what they currently have broken, so I wonder if status page services see some kind of ripple from github outages.
JsonDemWitOster 1 days ago [-]
About an hour ago I was having trouble browsing repo files in the browser and I thought "A disturbance in the force, is Github down?" Refreshed HN and loaded up their status site. Nada.
(Ofc, in a sensible universe, we just brush that off to a JS/Firefox glitch or my ISP.)
And yet, here I am. My code is not compiling, my AI isn't vibing, nonetheless I can't work! Two more hours before I can get off!
I've been against self hosting internal tools for a long time mainly because of the devops and other overhead. But AI based devops makes it so easy now to spin up whatever you want now that I'm reconsidering that. I use a lot of ansible for several of our deployments. At this point, most of that is managed via codex.
For Git, all you technically need is ssh access and some backup strategy for your server. It would be bare bones but workable. And there are of course plenty of OSS things that are a lot nicer than that.
I'm still using gh and gh actions and we are mostly below the freemium layer with that. But it is kind of slow and honestly a dedicated vm plus some high CPU/memory workers we can spin up on a need to have basis might be a lot faster. With GH outages becoming more common, my hand might be forced a bit.
In recent weeks, I've spun up listmonk (mailing list solution), matrix (as a slack alternative), and a few other things specific to our software stack. A github alternative would be more of the same. We don't need a lot.
The main objection is that with more moving parts to worry about, the workload for me also increases. Things need updating, monitoring, backups, alerting (and responding to alerts), etc. That sucks up my time and that is scarce.
Another reason for self hosting these days is that with agentic AI tools, self hosted things are a lot easier to integrate into agentic systems. If it is self hosted, you don't have to worry about API limitations, rate limitations, walled gardens, etc. All the traditional SAAS silos are becoming a problem from that point of view. The more locked down it is, the bigger the motive for moving away from it. That's why we ditched Slack for Matrix. Slack is hopelessly locked down and tedious to deal with. Matrix is super easy for this.
Why do they go down so often? Is it true that the reason is that they've incorporated too much AI without human review?
insanitybit 1 days ago [-]
It's (a) they're under massively increased load because everyone's vibing up new projects these days, (b) they've been in a weird frankenstein "on azure but also we have our own control plane" state for years and they're pushing to no longer have that be the case.
I don't think vibecoding at Github has much to do with it.
altern8 1 days ago [-]
Ah, yes. A lot more repos, commits, and most importantly huge PRs.
That makes sense. Thank you!
gilrain 1 days ago [-]
No, it doesn’t. Their competition is not similarly unstable, despite existing in the same world of LLMs. Think critically.
datsci_est_2015 1 days ago [-]
Devil’s advocate, Pareto heuristic would let us speculate that 80% of LLM traffic would be aimed directly at the largest provider, i.e. GitHub.
abejfehr 1 days ago [-]
I think it’s much more than 80%, it’s probably the default recommendation and folks who aren’t technical would just accept it. Probably closer to 95% or more
necovek 1 days ago [-]
Isn't the relative increase more of interest? If someone was only owning 10% of the market, and they've only gotten 8% (percentage points) of the 20%-not-GH LLM-related increase, they'd still be seeing a very similar spike compared to their baseline as GitHub.
gilrain 1 days ago [-]
Your speculation is that their competitors would naturally not see a commensurate increase in instability while “only” handling 20% of the same crisis?
I don’t buy the excuse. I want to hitch my wagon to those “mysteriously lucky” competitors. (And have. And haven’t had similar issues to Github, since.)
datsci_est_2015 1 days ago [-]
Competitors would be long tail, so a different mode of traffic entirely. Maybe they get spikes that are more easily whack-a-moled than the constant hammering that GitHub receives.
Tough to say as this is all speculative, though.
porridgeraisin 1 days ago [-]
It's probably a threshold thing isn't it? You wouldn't get 20% of the effect at 20% of the traffic. There's a step function in there somewhere.
vitally3643 1 days ago [-]
Their competition doesn't have nearly the same scale of traffic because they don't have nearly the same scale of users or network effects.
Think critically.
ModernMech 1 days ago [-]
I started using an agent (Codex) on my repo and it went from a a few dozen clones to thousands (3383 this week). I dunno what the agents are doing to clone the repo so many times -- I'm not running 3000 agents or prompts, maybe 10 or so this week. But if this is typical, a 1000x increase in usage across the board can't be good on the system.
12_throw_away 1 days ago [-]
> I dunno what the agents are doing to clone the repo so many times
agentic "ai" is going great
cautiouscat 1 days ago [-]
Microsoft has boasted 30% of their code written by AI.[1] However we could only guess if AI generated code is the issue or something else, or a combination of things.
That being said there was a noticeable trend starting around 2022.[2] That being said they’ve also been doing a big migration to Azure. It’s likely a combination of things.
This gets posted every time GitHub is down. This chart is not accurate. It is based on data scraped from GitHub's status page and that data is missing historical incidents from the pre-Microsoft era.
sarchertech 1 days ago [-]
Yeah, it’s not even consistent with their own incident history. I spot checked it and consistently found incidents with downtime/elevated error rates in months listed as 100.00000% uptime on that chart.
Gigachad 1 days ago [-]
The unofficial and offical charts are both lying. The GitHub one ignores actual outages and the unofficial ones count minor display bugs in minor features as a “github outage”.
sarchertech 1 days ago [-]
The unofficial one has done that for years though so it’s useful for comparison. If you go back a few years it was regularly at 99.9% uptime.
Gigachad 15 hours ago [-]
Just vibes wise, before Microsoft acquired GitHub, they added almost no features on a regular basis. These days they are adding tons of stuff every month.
When I dug in to the latest outages, they were almost all in small newer, features like all the AI stuff. The actual core GitHub platform seems much more stable than the unofficial uptime trackers propose.
sarchertech 4 hours ago [-]
I’m not sure about how stable it seems. I’ve been running into to slow or not working at all GitHub issues constantly over the last few months.
There’s also this blurb from a story on HN the other day that supports what I’m seeing.
“ I've felt this way for a long time, but for the past month I've kept a journal where I put an "X" next to every date where a GitHub outage has negatively impacted my ability to work2. Almost every day has an X. On the day I am writing this post, I've been unable to do any PR review for ~2 hours because there is a GitHub Actions outage3. This is no longer a place for serious work if it just blocks you out for hours per day, every day.”
cebert 1 days ago [-]
GitHub had a blog post about this recently. They reported a significant uptick in volume (repos created, PRs, etc.), which they attribute to AI usage and tooling.
Do you really believe their competition hasn’t seen the same increase? Because their competition certainly hasn’t seen the same instability issues.
abejfehr 1 days ago [-]
Yes, I truly believe that GitHub is recommended by an LLM orders of magnitude more frequently than any other forge
dawnerd 1 days ago [-]
I’ve interviewed a lot of people and when asking about their git experience they’ve said they use GitHub. To a lot of devs they are the same thing.
rwmj 1 days ago [-]
This plus in a well-designed system an increase in load might cause new jobs to stop running but shouldn't take down the whole system.
llbbdd 1 days ago [-]
What competition?
coreyh14444 1 days ago [-]
I personally trigger github actions approximately 50x more than I did prior to AI-driven developer coding and I'm not alone.
martinald 1 days ago [-]
Totally agree. There's days (or even afternoons) where I trigger more actions than I would have done in a month.
r0b05 1 days ago [-]
Okay so the recent outages are also likely due to increased load due to AI assisted development speeding up workflows.
AlienRobot 1 days ago [-]
It could be many things. Microsoft mismanaging stuff. Azure. Vibe-coded Github. So much AI slop being committed it adds an extra burden on the servers, etc.
pistoriusp 1 days ago [-]
Whilst you're waiting for it to come back, try out AGENT-CI (which is a project I built.), which runs GitHub Actions on your machine: https://agent-ci.dev. (Open source, etc.)
No, it's not like "act," because it uses the standard Github runner, the difference is that the control plane is an emulation of api.github.com, because of this we can do all kinds of nice things:
Caching in ~0 ms. Pause on failure, so you can let your AI agent fix it and retry without pushing.
skinfaxi 1 days ago [-]
You're affiliated with the project. You should definitely be upfront about that when shilling.
pistoriusp 1 days ago [-]
You're right, figured it was implied, but now fixed.
ramon156 1 days ago [-]
"Its not like act, because we can add AI"
Is what it boils down to.
> codex "Fix this pipeline, use `act` to verify your changes"
pistoriusp 1 days ago [-]
I did not say that, what I said was: It's not like `act` because it's not a rewrite of the runner. It's the standard runner... So the one that actually runs GitHub Actions.
I have tried to use act many times, and many times I've failed.
P.S. pause on failure is also helpful for humans, but I'm trying to be realistic about where the future of programming is going...
Xirdus 1 days ago [-]
I had extremely bad experience trying to setup act on my Macbook. If this is something that actually works (and doesn't steal my credentials), I'm willing to try it despite AI non-features.
Groxx 1 days ago [-]
Yea, I've had only barely-success on only a few projects with act. Usually due to steps/scripts that use github-internal APIs, but afaict far from always.
I like that it exists, but what a freaking mess that it's necessary and so difficult to do.
pistoriusp 22 hours ago [-]
Please try agent-ci; it's github actions that run locally. Nothing less.
a1o 1 days ago [-]
What I don’t get about this is how you run OS specific tasks (Windows, macOS, Linux)..
I started playing with proxmox VMs and containers in them (docker and tart) to see if I can build some local infrastructure to properly solve this…
pistoriusp 1 days ago [-]
We support macOS via tartlet, but basically it's always linux. If you need windows then it's gonna be an issue.
The jobs runs via containers.
thepaulmcbride 1 days ago [-]
We’ve had GitHub actions for long enough, it’s time for GitHub consequences.
cyanydeez 1 days ago [-]
>Copilot: Do you want me to implement consequences for you or babble on and on about what might entirely be a figment of your imagination (Github is up and you're on a 48 hour bender without sleep)
pnvdr 1 days ago [-]
i would like to see consequences for "secure sleep" XD.
shevy-java 1 days ago [-]
I think this can only happen if there are viable alternatives.
For instance, the UI at setups such as https://git.devuan.org/Daemonratte/gtk2-ng is quite ok-ish, in my opinion. Granted, it is mostly copy/paste from github but that still is about 1000000x better than sourceforge's interface - and gitlab's UI too (I just hate gitlab's UI, they seem to love complexity and a billion features only 0.000001% ever need; GitHub, with all its faults, is for the most part really simple - not everywhere, e. g. GitHub wiki setup sucks, but by and large I think it is simple overall).
danudey 1 days ago [-]
Github definitely has the better UI but if it weren't for network effects I'd be pushing to migrate to Gitlab pretty hard.
Bnjoroge 1 days ago [-]
Gitlab’s UI is extremely terrible. It’s hard to even explain how bad it is.
dilawar 24 hours ago [-]
Once you get used to it, it's not so bad which is probably true for all functional UIs. I switch between gitlab and GitHub quite a lot and I can't say which one is objectively better. I do like that cross-linking is easier in GitHub but I prefer gitlab ci over GitHub actions. Too bad that gitlab ci runner has removed the command to run ci locally but third party foss solutions are there.
wongarsu 24 hours ago [-]
If replacing github wholesale isn't viable, how does the story for replacing GitHub Actions look like currently? I don't remember the pre-Github-Actions days of everyone using CircleCI with a github integration in a negative light. I've noticed that since then a couple of CI providers have sprung up that differentiate themselves with faster build speeds, but I haven't really kept up with that market
joshuanapoli 24 hours ago [-]
I've been thinking of reverting back to Circle CI.
thesurlydev 24 hours ago [-]
"A little less conversation, a little more action please"
mimsee 1 days ago [-]
Apparently they deleted the Github Actions account as it shows up as ghost in PR comments.
rschiavone 1 days ago [-]
if that's the case, such a thing is so absurd that it goes around and it becomes almost hilarious
this_user 1 days ago [-]
I would love to know how much of their internal workflows are being handled by AI workflows. Because this seems like the kind of thing your agent might do.
DoctorDabadedoo 1 days ago [-]
The team got greenlight to more tokens and the problem should be fixed soon. Fingers crossed. /s
source: voices in my head. Not affiliated with MSFT.. anymore.
kminehart 1 days ago [-]
Are there any GitHub Actions-compatible CI services out there that don't rely on their infrastructure? I know of depot's but no others; are these resilient to these outages or do they still lose functionality? I imagine the latter but I don't know.
4lun 1 days ago [-]
We currently use external runners (Blacksmith.sh), but that didn't shield us from this as GitHub actions is still the control plane for triggering and monitoring them.
We're now considering Buildkite (apparently they have a GH actions migration tool) or self hosting something (GitLab CI, maybe even Jenkins), as it looks like that would've kept ticking over since we're still seeing webhooks being triggered today during the downtime.
kylegalbraith 1 days ago [-]
Try Depot CI as well. Supports a GHA syntax but the entire control plane is ours with our own engine.
kylegalbraith 1 days ago [-]
Founder of Depot here. To my knowledge, we are the first engine to support different syntaxes in this compatible way via Depot CI [0]. Great time to try it out and let us know your thoughts! We’ve built a lot of cool stuff into it like parallel steps, custom images, and a full CLI/API interface so you can literally everything without going into the web app.
Is there a tier for open source organizations? Do I have to admin any of AWS that runs behind the scenes or can I pay a fixed price to depot and get it to solve everything out of my way?
I used to use Cirrus CI as an alternative to GitHub Actions and am looking for a new alternative. I wonder if Depot could fit in the same way for my needs. I need to run builds and tests in Windows, Linux and macOS.
heeton 1 days ago [-]
As someone who partially uses depot but was still affected by this github issue, we obviously haven't moved over enough. We use your runners but github is still blocking us.
Hope you don't mind the public ask, it seems useful for others.
If we're using depot runners, and want to use them directly, or move off of github actions being the controller for when things run: what do you suggest?
Trigger the workflows directly on depot via CLI?
kylegalbraith 1 days ago [-]
Yes, triggering Depot CI via the CLI is the sure fire way to avoid all dependencies on GitHub.
We’d need more details around what you’re seeing. It is true that if auth across GitHub is broken than we can’t copy your actions out to be used by Depot CI. However, we have a solution in the works for that as well.
In short, Depot CI, our own engine and control plane is not dependent on upstream actions control plane. But still has to listen for commit events to know if/when to run jobs on things like PRs. This to is being removed in the future.
kevinminehart 1 days ago [-]
Are you able to bring your own runners? Our org is heavily invested in self-hosted runners at this point and have gotten a pretty tremendous value from it. I think we'd be wise to get away from GitHub's control plane but keep running jobs in our own infra.
kylegalbraith 1 days ago [-]
Yes, we support this via Depot Managed for all of our products including the latest one: Depot CI [0].
github actions themselves can be self hosted, its quite nice actually to be able to keep your same patterns as cloud hosted actions and with one line change to the yaml have it running on your own hardware. I do this for actions that take 6-7 hours so I am not burning through the 3000 minutes that come free with my account.
mdrachuk 1 days ago [-]
Self-hosted action runners are not working too right now.
kminehart 1 days ago [-]
This isn't resilient to this downtime though. Our self-hosted runners are currently not functioning because of some github dependency.
asimovDev 1 days ago [-]
what kind of actions take that long? some kind of compilation task / gigantic test suite ala SQLite?
ttouch 1 days ago [-]
there are a couple and have very good reputation - though I've never used them
They also say that they're much cheaper than github
kevinminehart 1 days ago [-]
I think both of these provide nodes that are scheduled using GitHub's control plane. They would also not be working right now.
Onplana 1 days ago [-]
Is it about funds? Why Github is not catching up with the traffic? I know there's a mass rush on Github recently specially due to Claude Code leading users to use Github. sometimes even persuasive.
discordianfish 1 days ago [-]
Because scaling complex systems is not trivial
surdu 1 days ago [-]
It was pretty easy before October 2018, when Microsoft bought them:
Scaling to human use vs automated/agentic use is a very different thing.
gbear605 1 days ago [-]
Github has had horrible uptime for years, before agentic use was a thing. The killer was Microsoft.
surajrmal 24 hours ago [-]
It had uptime issues before Microsoft. They just got better at tracking and reporting after Microsoft.
surdu 4 hours ago [-]
So, before October, they were lousy at tracking downtime issues for 2 years (no downtime from 2016 to 2018), but in November, Microsoft came and gave them the technology to correctly track downtime, and they had their first downtime logged in November.
Is this your argument?
gbear605 19 hours ago [-]
My subjective experience was that it got a lot worse post-Microsoft. I could be misremembering though!
Barbing 1 days ago [-]
Do we know this is the predominant reason?
discordianfish 1 days ago [-]
Not saying they are doing a particular good job but its not as simple as "paying more for bigger machines" and be done with it.
AlienRobot 1 days ago [-]
To be fair, that was 8 years ago. Github now has all that days and users + 8 years of data and users.
Aperocky 1 days ago [-]
Sometimes it is. There are some incredibly brute force yet simple and elegant pattern that power some of the biggest scale system you could think of.
It is relatively easy to scale a collection of simple things to extreme and exhibit complex behavior together. It is a lot harder to scale something complex to extreme. But too many times the latter is the default - designed wrong from the ground up and stuck in scaling hell.
Barbing 1 days ago [-]
They’re focused and hiring right and managing right, but this is just so difficult it’s bound to go down?
If Google owned GitHub would they be better positioned to scale?
chocrates 1 days ago [-]
Someone said GitHub is racing to the mythical "zero nines of availability" and I love it
Andrex 1 days ago [-]
Hmm... 88.8888888%?
Jesus, that's both horrible and seems within reach.
Miner49er 1 days ago [-]
They've already been well below that over the last 90 days
LorenDB 1 days ago [-]
Yep, they just need to improve their reliability by 2%!
This page tells a very different story from GitHub own status page. What is different here?
alexfoo 1 days ago [-]
Github measures/reports the SLA of the individual services.
The external page linked above goes the other extreme and considers it a bad status whenever any individual service is degraded.
In reality the majority of people only use 3 or 4 of the core services the majority of the time but since there's no "core services" SLA/uptime the usability of github for the majority of people is slightly obfuscated.
JCTheDenthog 1 days ago [-]
Part of it is that it considers downtime in any of the services GitHub provides as GitHub being down. So if GitHub had 100 different services, and only one of them was down at any given time (but at least one was always down), then it would show 0% uptime.
0xbadcafebee 1 days ago [-]
If you want an alternative to GitHub Actions, you could self-host Forgejo Actions, but I'm not that happy with the design.
I much prefer Woodpecker CI, which is an open source fork of Drone.io. It supports multiple Git backends like GitHub, Gitea, Forgejo, Gitlab, Bitbucket. It supports running jobs locally, on Docker, and on Kubernetes. And there's autoscalers built in for AWS, Hetzner, Linode, Vultr, and Scaleway. There's a bunch of 3rd party plugins (https://woodpecker-ci.org/plugins) for custom integrations. The UX is also very simple, with OAuth used not only for authentication/authorization but also setting up & accessing repos. The system architecture is great, with separate components that run stateless connected to a database, and a custom plugin is any program that takes environment variables and does stdio. The config file is a good balance of ugly YAML and convenience syntax like shell-style parameter expansion variables.
It probably takes less than 15 minutes to install, set up, and run WoodpeckerCI for a small team, so it's not a big investment to try out or host. With the autoscaling plugins it lets you scale your workload up to whatever size. Honestly you could run it on a laptop since it's written Go.
If you don't want to self-host Gitea/Forgejo, I recommend SourceHut for private repos and Codeberg for public ones. Happy to answer any questions you might have for either based on my experience!
packetlost 1 days ago [-]
For private repos I just have a folder on my NAS that I run `mkdir <repo name> && git -C <repo name> init --bare` in. Works great.
paularmstrong 1 days ago [-]
What's wrong with codeberg for private repos?
pdpi 1 days ago [-]
They only allow private repos as an exception, and only insofar as they're ancillary to open source projects.
From their FAQs[0]:
> Codeberg's mission is to promote free/libre software. Keeping software private is obviously not our primary use case, but we acknowledge that private repositories are useful or necessary at times.
I'm more than happy with https://codefloe.com for private repos. The service is blazing fast and the maintainer does a great job keeping it up-to-date with recent Forgejo releases.
dsco 1 days ago [-]
Yeah I'm getting an error where it says account has been suspended. They really are becoming an embarassment
eatyourpeas 1 days ago [-]
this has happened to me too. i am guessing then it is not a real reason?
maratc 1 days ago [-]
`github-actions[bot]` was disabled for some time, if that's the actor which does the checkout in your setup it could be related. FWIW it's back to working now.
wg0 1 days ago [-]
I think Anthropic should buy Github after buying bun and everything in between.
KaiserPro 21 hours ago [-]
Indeed, they are the masters of keeping a service up all the time
FelipeCortez 24 hours ago [-]
so they can rewrite it in Rust?
vyrotek 1 days ago [-]
It's crazy to us how Github Actions have these issues but Azure DevOps never has these hiccups for us even though we hear they're on the "same infra". We're happy to stick with DevOps.
sgerenser 24 hours ago [-]
Who says Azure devops is on the same infra as GitHub? I mean, sure they're both hosted in Azure data centers, but there's very little else shared between them AFAIK. I used to work for Microsoft and I heard about the grand plans to merge the two but I don't think it ever really happened.
vyrotek 21 hours ago [-]
It's come up here in a few comments before.
Perhaps not 100% physically shared infra but there's references of architecture overlap such as "The GitHub-hosted runner application is a fork of the Azure Pipelines Agent."
This is outrageous. Someone go create a Polymarket.
LorenDB 1 days ago [-]
Please don't. These "prediction markets" are a scourge upon mankind.
smilespray 1 days ago [-]
How to kill a business 101. The brand damage to business and owner is incalculable.
ElFitz 1 days ago [-]
I recently switched from GitHub Actions to Buildkite + self-hosted runners.
Setting it all up would have been tediously annoying eight months ago (Buildkite requires setting up GitHub webhooks for each repo).
Last week I just had codex set up everything, ephemeral vm runners and all, using a couple of low-spec refurb mac minis, Buildkite’s API, a short-lived API token, and migrate my repositories one by one.
So far so good, it’ll pay for itself within two to three months, and following today’s outage I suggested at work that we experiment with the same set up.
They’re considering it.
ibejoeb 1 days ago [-]
What problem is github solving that has led it to become critical infrastructure for so many? Is it that everyone is remote and VPNs are too much of a hassle to give everyone access to a build server? Is the serving as the authoritative auth for development services? Does it provide better compliance reporting? It just isn't apparent to me what github offers that you can't get elsewhere with at the same cost and effort. I've been in some pretty large orgs with distributed personnel, but this just hasn't ever been a problem.
Xorlev 1 days ago [-]
GitHub solved the original "code collaboration" problem, and now it's a default easy way to outsource repo management. It also has the most integrations. A lot of companies grew up using GitHub.
GitHub was, once upon a time, quite stable. Things have changed: more features, more usage, and automated agents.
ibejoeb 1 days ago [-]
I know what it does, but why is it such a problem that Actions is down? I think you did kind of answer it: "A lot of companies grew up using GitHub," i.e., they are using it as infrastructure by default, not because it does something that otherwise can't be done.
repeekad 1 days ago [-]
It’s well integrated into massively underpriced agentic coding (and noncoding) workflows, I doubt there’s much more reason than that. The hip thing to do now is hold all your docs in github instead of notion so your agent can traverse them locally
vervas 1 days ago [-]
GitHub Actions is the build server. You could use any other but it is convenient indeed to have it integrated in your repository hosting service.
BrunoBernardino 1 days ago [-]
If you don't want to self-host Gitea/Forgejo, I recommend SourceHut for private repos and Codeberg for public ones. Happy to answer any questions you might have for either based on my experience!
delf 1 days ago [-]
If you would like less dependence on GitHub for issues and PRs, please check out GitSocial, it stores everything in git itself, making them portable and offline-first.
I've started spending each github outage planning our move to an alternative. I guess I'm not alone. Where are you all moving?
Mashimo 1 days ago [-]
We use TeamCity for CI builds, before that Jenkins. Only accessible from the inside of the network.
Even though it's selfhosted and we don't have a dedicated infrastructure team, I don't remember it ever being down in the last 12 years I have been working here.
trollbridge 1 days ago [-]
I switched to GitLab a while ago and then spun it up locally.
Something’s wrong when my own infrastructure is more reliable than Microsoft’s.
stuff4ben 1 days ago [-]
Let us know when your infrastructure sees the load that Microsoft's does and how you've handled it.
trollbridge 17 hours ago [-]
The entire premise of hosting locally is you control the load.
r0b05 1 days ago [-]
It's so weird because github used to be known for rock solid stability and now the entire reputation has changed.
fooster 1 days ago [-]
You must be new. github was never that stable.
parisiansam 1 days ago [-]
free service is down again, let's everyone that use the service for free complain again!!! (sorry for the sarcastic comment but i find it crazy how people feel they are entitled when it's free)
EDIT: sorry i meant this rant at the one complaining for the free service not for the paid customers (which is unacceptable)
gloosx 1 days ago [-]
If you take a bit of a closer look, github.com has a "pricing" page
Hey at least Copilot AI Model Providers have 100% uptime, so there's that
comboy 1 days ago [-]
I have fun somebody imaging somebody internally explaining that this is a heavy traffic page and we should use it to increase reach.
katss 1 days ago [-]
What could be the cause of GitHub issues from an engineering perspective?
emirhanerkan 1 days ago [-]
In my mind there's no doubt Github datacenters can't handle the recent load that came after agentic AI. They just need to get new servers. It's simple as that.
qwerpy 1 days ago [-]
“Just” “simple as that”
Reminds me of the occasional “JavaScript developer tries to vibe debug a Linux kernel issue” comments we get here.
tux3 1 days ago [-]
Over the last 90 days the status page (https://www.githubstatus.com/) shows around 2 nines of uptime for most services.
I trust that about as much as I trust Volkswagen's emissions numbers.
ripitrust 1 days ago [-]
I initially thought it was because I ran out of action minute, and was about to upgrade my plan
Lucky I came here before hitting the confirm payment button
amirhirsch 1 days ago [-]
Shout out to all my SF 5am crew checking if their overnight prs passed CI. Real 597 “member of technical staff” energy. I guess we should expect this, it is a Tuesday!
sibidharan 1 days ago [-]
Had to figure out it was Github and not my AI Agent... Sad it got scoldings for being lazy on waiting for CI checks! What a waste of tokens!
timedude 24 hours ago [-]
Aws made a very big mistake to stop their CodeCommit service. They could have eaten Githubs lunch if they had continued to build it out
1 days ago [-]
1 days ago [-]
stevenhubertron 1 days ago [-]
This is great because I finally set up Actions yesterday for a new project of mine and of course it’s failing today and thinking I screwed up the yaml.
mghackerlady 1 days ago [-]
I don't understand anyone still using github for anything unless they have to or have payed for it. Move literally anywhere else
devil1432 1 days ago [-]
I wonder if these github failures are just systematic incompetence or MS cutting budget on purpose to promote its own cicd tools
mattbrewsbytes 1 days ago [-]
Or possibly an elevated number of AI Slop Cannons aiming their LLM generated hallucinations at github hosted repos?
couAUIA 1 days ago [-]
LoL they added "Copilot AI Model Providers" in githubstatus and it has 100% up time.
Thanks for pointing out that nobody is using that thing
gib444 1 days ago [-]
List of things "DoS"d by AI:
- GitHub
- Hiring budgets
- RAM (/personal computing in general)
- Electricity
- Media/Content
- Truth
shwetanshu21 1 days ago [-]
And it is bypassing mandatory GHA Pipeline check and giving green. So be careful when merging/reviewing your PRs cause.
liamdoyle 1 days ago [-]
Has anyone actually moved off? If so where?
I like being able to vote with my (teams) wallet and I'm tired of staying out of convenience
rebolek 1 days ago [-]
I moved to Codeberg and self hosted Forgejo. I'm happy.
ceheaaf 1 days ago [-]
What's that? You're still using microsoft products? Guess that's your own fault.
cebert 1 days ago [-]
I think we should start betting if GitHub will be down on Polymarkets or something at this point.
fidotron 1 days ago [-]
The future of SRE will be the company putting some amount of money on a prediction market against the site going down and you get to take home the winnings as long as the site stays up.
mattaustin 23 hours ago [-]
Running a company without a CEO is really starting to show..
sh-cho 1 days ago [-]
'Degraded' should be banned in status pages. It sounds just irresponsible, like "Yeah, it can be slow or something sometime. Whatever. Who cares"
Andrex 1 days ago [-]
Straight-up, "degraded" should strictly mean "may be slower, or so slow it randomly fails" on these kinds of status pages.
bobmcnamara 1 days ago [-]
The whales are all dying, and we don't know why. Well, some are still alive for now though so maybe it's not so bad...
jaapz 1 days ago [-]
How would you call "available, but only sometimes"?
hartjer 1 days ago [-]
Going to need to make an isitup website for it soon enough
CSMastermind 1 days ago [-]
Will more copilot usage fix this? We should try more copilot.
tom1337 1 days ago [-]
no maybe we should make copilot the pilot so the bad humans in the loop finally cannot break anything.
mohsen1 1 days ago [-]
oh man spent so much time trying to debug what's going on. I have a complex setup with GitHub Actions and self hosted runners so I thought it's something broken in my CI setup
heeton 1 days ago [-]
Ugh, same. 30 mins with 2 devs trying to figure it out before they posted an update.
At this point, GitHub should rename itself downhub.
hansmayer 1 days ago [-]
No way - everyone tells me the AI adoption is going great?
nickstinemates 1 days ago [-]
The main operating model with git is going to go back to decentralized. Setting up and managing something like https://forgejo.org/ is a way better experience than constant interruptions by a faulty service that can't meet demand.
The open source contribution model as we once knew it is dead; you're not going to accept patches from random agents. The risk is way too high. And you can see that increasingly "AI Slop" makes it difficult to be a maintainer of any semblance of a popular repo.
So what's the value? A durable place to store work? hah.
Discovery? That part of Github has always been shitty.
So that leaves.. Github Actions? The thing that is down every other day and has been the subject of a few ~rug pulls~/attempted price hikes that are almost surely coming back?
danieloj 1 days ago [-]
Does anyone use any good alternatives to GitHub Actions?
1 days ago [-]
miki123211 1 days ago [-]
This is your periodic reminder that Github is growing at ~14x (1400%!) annually. This would be incredible growth for a young, unprofitable, VC-funded startup, even Uber never achieved more than ~3x AFAIK. For a widely-established company that was already very well known and a market leader in its niche for many years? Absolutely unprecedented.
This is a conservative estimate assuming linear growth, the actual number is likely going to be higher. Much higher.
It's not too hard to grow 14X YoY if you start from a hundred customers. If you have hundreds of millions? Yeah, not so easy.
Growing what exactly. Seems like a spam issue or some sort of automatic circular commit chain between alot of projects.
adamddev1 1 days ago [-]
How's the AI generated code running for ya?
toobulkeh 9 hours ago [-]
At some point I’ll need a status page for when it’s up
markfsharp 1 days ago [-]
Contingency action plan: Codeberg. Engage.
throwatdem12311 1 days ago [-]
When is it up?
SideburnsOfDoom 1 days ago [-]
Github is more likely to be up before noon in UTC timezone. i.e. before the majority of US users are online and causing load.
Or maybe it's before the GitHub internal devs are online and deploying changes.
carreau 1 days ago [-]
i still can't see many pull requests in a bunch of repositories... it's been over a month
rock_artist 1 days ago [-]
Super odd make productivity useless
shevy-java 1 days ago [-]
Microsoft is really working hard to kill off GitHub now. That's quite amazing.
We have already seen this in the last some weeks, but now this has become a meme that keeps on giving. GitHub down! GitHub up again. GitHub Down! GitHub ... ...
hmmdog 1 days ago [-]
Tell Claude to fix it, simple.
Cupprum 1 days ago [-]
It should be up again
jamie_davenport 1 days ago [-]
This has become so typical that we've started working on a modern Github alternative called Plain.
My first time using GH Actions was last week. GH was so flaky that pulling a submodule failed >50% of the time. I had to write a script to retry pulling the submodule in a loop.
I've done some hacky shit in CI scripts, but none made me more mad than that one.
suis_siva 24 hours ago [-]
Not to shill myself, but I'm sick and tired of this and been sick and tired for the last month. Decided to quit my job to work on https://harmont.dev
shadowbip 1 days ago [-]
here all is ok, 3 actions without problem
hxii 1 days ago [-]
At this point it's as if the team there went "fuck it, let's just watch it all burn down" or something.
With all the recent negativity – how are they not even TRYING to fix the damn thing?
saltyoldman 23 hours ago [-]
Git is decentralized until it's used by companies.
bdangubic 1 days ago [-]
Feels like Github Actions is UP should on the front page (when it happens) at this point. Down is no longer front page worthy
dncornholio 1 days ago [-]
Stop relying on Github.
Self hosted Gitlab with self hosted (or AWS) runners running your pipelines.. We only use Github as a mirror for our public repositories.
j45 1 days ago [-]
With the increasing challenges from bots and ai agents created with toddler level clarity, Self hosting is going to continue to work.
moonrailgun 1 days ago [-]
my work is totally stop. cry
sylware 1 days ago [-]
microsoft github should work at restoring interop with noscript/basic HTML browsers...
matt_kantor 1 days ago [-]
I agree, but that's not at all related to this outage.
5 hours ago [-]
sylware 1 days ago [-]
Yeah, just reminding people here about that.
I am trying to refrain my "off topic" rants... but such microsoft github abuse is generating so much hate due to their dominant market position, it is hard.
booleandilemma 1 days ago [-]
Too many DEI hires? Or maybe H-1Bs? Or maybe it's a vibe coding problem.
throwawaypath 1 days ago [-]
Looks like removing the "meritocracy" doormat, hiring Coraline Ada Ehmke, and changing "master" to "main" paid off in spades!
aa-jv 1 days ago [-]
Too many times we've been bitten by this - it has been an issue too many times to count.
This is why we don't use Github Actions, kids.
Seriously, its a proprietary build service that puts the keys to the kingdom in someone elses' control. Just: No!
Print this status page to PDF so you've got it handy next time someone castigates you for not using Github Actions, folks.
vucetica 1 days ago [-]
So, what do you use?
aa-jv 8 hours ago [-]
Local CI, duh. Its the only sane approach in this day and age. Handing the keys to the kingdom for someone else to build and maintain it is just nuts.
blinkbat 21 hours ago [-]
what else is fuckin' new.
rvz 1 days ago [-]
Another outage at GitHub with actions and pages not working thanks to the AI agents Copilot and Tay.ai creating more issues. Last time this happened was 6 days ago. [0]
This time today it was caused by friendly fire by the automatic suspension of the GitHub Actions bot which is now a "Ghost" user. Since there is no CEO of GitHub to contact it we are just going to see more [1] of this again.
You might need to push a critical change soon, but now you cannot. You won't get any of these issues if you self hosted as I said 6 years ago...[2]
I commented on the other post, but GHA's awful reliability, ergonomics and performance have caused me to quit my job and work on https://harmont.dev.
gustavus 23 hours ago [-]
Anyone else notice that the first/near top comment on every HackerNews post lately is someone saying something along the lines of
"I had X problem so I went and started working on Y solution if you want to give it a look?"
I don't want to delve into it any further - but something about it seems incongruous. It's not spam it's submarine marketing.
sunaookami 20 hours ago [-]
It's called growth hacking and it's indeed rude and annoying.
suis_siva 22 hours ago [-]
Ah sorry! Didn't mean to be annoying. Hm is open-source and my intentions are good. I'm also trying to figure out what's good and what's not!
Apologies for the spam!
weakfish 23 hours ago [-]
I mean, it is a community that self-selects for builders and startup-types.
Which certainly made me shit myself, briefly.
The day it broke away and became centralized was when we had a PR + mandatory "Required actions" to merge to main.
https://reticulum.network/manual/git.html#mirroring-reposito...
Just set up a Kubernetes deployment and you’re set.
But as others mention, GitHub’s primary strength is collaboration. If you want decentralized, solve this by creating a decentralized collaboration tool on top of fossil and/or git.
For example, how to do pull requests and code reviews?
Gosh, it's hard figuring out what changes Lorne made if only we had a system to merge those changes. Enter git
Gosh it's hard figuring out what packages Rachel had to make this work. Enter rubygems/pip/npm
Gosh it's hard figuring out sync these changes across a network. Enter github
Gosh it's hard figuring out how to get those packages working on my operating system. Enter docker
Gosh centralizing our distributed version control software system onto one website is getting really unreliable. Enter fossil(?????)
If we go any further having one computer per business with a sign up sheep is starting to sound pretty fucking attractive.
being a host for git repositories has never been its core competency. neither has its groupware offering.
does it even serve OSS well? a very interesting criteria is, "Have mature or adopted end-user-facing OSS recently merged a large PR from an unallied contributor?" The answer is overwhelming no. This is why there is so much innovation in this space.
Proudly self-hosting Forgejo since then.
> Our team is currently experiencing an unexpectedly high volume of tickets which has resulted in longer response times than we prefer. We acknowledge the long wait and apologize for the experience.
> Sometimes our abuse detecting systems highlight accounts that need to be manually reviewed. We've cleared the restrictions from your account…
Fully self-hosted IMO can be an overcorrection. The issue isn’t “relying on other people”—it’s relying on GitHub, when they’ve made it clear they don’t care about uptime and they don’t care about support turn-around-time.
It would be a pain as I'd have to set up a few integrations again, but github is far lower down the risk scale than the vast majority of SAAS providers
I hope people here are aware that you can push your repo somewhere else if wanted.
Git is a distributed system, there isn't even a server, only other git repo instances that are remote.
It will be a hassle to migrate to another platform, possibly a couple of hours work to do the 25 repos in my ~/git/ directory.
Even highly complicated actions can be migrated quite easily -- the source is stored in .github/workflows/blah.yml
https://www.youtube.com/watch?v=LGeOee7x5lY
Is it true that official service status pages are updated automatically?
Depends. Typically no because there’s an art to crafting the actual message around impact… but sometimes yes it is automated
I was thinking more of needing to notify/get sign-off from management...
If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well.
But effective monitoring is harder than people assume.
Isn't that what monitoring actually is? The issue seems to be in their testing, not monitoring.
There are synthetic tests, where you can generate API request calls or even simulate an entire user journey. These allow you to control the user agent, the payloads, and thus you know anything errors back are actual errors. These are triggered by the observability platform (think like running a cron-job) and thus you're not tied to user activity to see when problems arise.
There are other metrics outside of HTTP response codes too. Think like free RAM, CPU usage, disk space, etc. This is just naming some obvious ones because these types of metrics are generally bespoke to the type of application your monitoring. And with these types of monitors, you'd not just have an alert when things have failed, but ideally have alerts when an irregular trend is showing that things are likely to fail too. This latter type of monitors helps you get ahead of the problem before it become customer facing.
Then you have more traditional stuff like logs. This will also be bespoke to the application. But you'd expect errors in logs to get surfaced quickly. Assuming Github have good hygiene in what's being logged.
Tie that up with APMs, RUM, and other goodies like that and you'll have diagnostics to investigate issues when they appear.
(this is just a super high level view of observability too)
You should not alert on cpu, ram, etc
It doesn't "need" that. That just how most people set it up because it’s an easy sane default that allows for network jitter without inexperienced engineers thinking about different conditions triggering different types of responses.
If you’re measuring internal APIs from an observablity solution that’s has nodes already inside you’re network enclave, then there is a strong argument for alerting early.
> You should not alert on cpu, ram, etc
That’s not true to say as an absolute statement. And a generalisation it heavily depends on the system your monitoring and how it behaves under pressure.
But in any case, I wasn’t suggesting CPU alerts were the end goal. I said:
> these types of metrics are generally bespoke to the type of application your monitoring.
Ie you’ll use metrics but those metrics will be highly specific.
The CPU examples were an illustration as to what a “metric” is (it might seem obvious but not everyone is an expert) but the point was HTTP response codes aren't the only types of metrics one should be capturing and watching.
If your requests are fast and cheap, you can probe frequently relative to your goals, but often that's not really possible (think, long SQL queries, or scheduling a container/pod). There you need several datapoints, or possible fewer augmented with other signals.
Talking about long SQL queries, I quite like throwing CPU alerts on database servers. They'll be a low priority alert (ie no out of hours "pagers") so just something that goes into a slack channel. But they're a good indicator of when developers have poorly optimized SQL, or the DB schema is poorly defined (eg missing indexes), or the DB server itself is poorly sized.
This wouldn't be something you'd expect to need in production and definitely not something you'd rely on as a notice of a production outage. But it is an example of one of those 1% occasions where a CPU alert does add value to the overall observability of the application.
But this also ties into your excellent point about how you'd use CPU and other data points to build a picture of what's happening in your application.
idle CPU is often wasted CPU
Who says public status page equals internal monitoring.
They likely know faster than you. Whether they post it publicly is a different issue (hint: SLA penalties, news impacting stock etc)
Are you sure you’re replying to the right comment?
For context, the parent comment you replied to started with status page.
Then are you talking about internal leaks or just guessing? Otherwise besides what's public how do you know they don't know?
Someone then replied about how it takes a bunch of HTTP response errors for problems to be alerted and thus I commented that application observability would consist of more than just waiting for users to hit errors.
Maybe the Github Actions infrastructure isn't run like that.
edit: my oncall rotation notified on all 500s, 24/7, not just rates - https://news.ycombinator.com/item?id=48279262
Recently there was this: https://news.ycombinator.com/item?id=47252971 "10% of Firefox crashes are caused by bitflips"
Which makes me think a small amount of random issues which happen even though nothing is broken, is normal everywhere. Especially once move things around on a network, there's potential for a lot more random errors.
It does require constant tuning and adjustment though.
This is why data hoarders who have NASes with lots of space insist on running their servers with ECC RAM despite it being significantly more expensive. Because bit flips, for all intents and purposes, cannot happen. The RAM itself detects and corrects for them.
I wouldn't expect bit flips to be a significant contributor to enterprise problems.
If your network goes down because of a DDOS, or part of your system overheating, that's an internal issue you had control over.
If a bit flips because of cosmic radiation, you can't really do anything about that, and it's utterly unpredictable. That's "random" to me.
I know all of Gmail, every GCE service I can think of, every AWS service I can think of, Amazon.com, Netflix, and Github all do not page on just a single 500.
I know none of those are particularly "high performance" though. Curious where your experience is coming from.
I had a fairly long tenure, where I maintained multiple key services in critical online payments flow. Authentication, authorization, core business and risk data, as well as some cross-cutting control plane stuff, etc. You needed one or more of our services to take a payment, serve any request from the employee dashboard - pretty much everything hit our services. The entire company ground to a halt without my team.
We paged for every single 500. In instances where a particular class of 500 was spurious or not worth fixing, we would leave it acked or mark it as noise. But typically we'd just put in a fix as soon as possible so we didn't page.
Our graceful shutdown and traffic shaping stack was great, but occasionally we'd get a few pages during deploys or failovers.
Oncall was typically not bad, but when it did get bad it was terrible. I've been involved in huge outages that cost hundreds of millions of dollars. Usually it was the fault of multiple teams having compounding runaway failures rather than one service or bug in particular.
It's inexcusable to have a customer's payments not go through. We engineered around resilience. We had strict five nines SLAs and p99 targets and evaluated our adherence with even the smallest partial outage. Hundreds of other services depended on ours, and downstream impacts were huge, so we had to keep a tight ship.
We didn't have "business hours"-only paging either as our platform was available globally, including a heavy install base in Asia.
Assuming the existence of some kind of network (with zero guarantee of 100% reliability), how does this work in practice? Is each 500 treated as an event that needs investigation, even if the result of that would end up as 'a router dropped something from an internal buffer but the transaction as a whole was re-tried by a parent so the service itself recovered'?
https://youtu.be/zR9PpXWsKFQ
Even if it's "DB in datacenter I tried to save to was hit by meteor" event, you can cater for this not to result in 500 (ie - DB unreachable, retry in a couple of minutes); the question is if you want to.
If my DB health check endpoint is returning 500s for N consecutive checks over M minutes, yeah, please wake me up at 3am!
If one user hit a weird edge case in form validation and got a one-off 500, please don't! We can fix that on Monday.
Not always easy to distinguish those clearly or configure those business hours rules, but for my team at https://heyoncall.com/ that is the goal -- otherwise your team burns out fast. Waking up someone at 3am has a real cost, so you better be sure it's worth it.
As others have said, follow-the-sun type models do exist, usually staffed by people in their normal working hours (EMEA, Americas, APAC) but this means you've still got to cover the weekend and public holidays (which there are a lot of when you factor in plenty of different countries).
Where you need a quick response you can have a core ops/noc team that looks at things with lower thresholds and shorter windows, and their job is to do the initial triage and then page the appropriate team earlier than they would have been alerted by their own alert thresholds/monitoring.
Actually clicking the button to change the status on a public status page is a whole different topic that becomes very political in certain companies.
But if it is synthetic queries sent from the monitoring platform, then you control the user agent, payload, and endpoints. So any failed requests are a symptom of a misconfiguration and/or failure that should be investigated. Albeit not necessarily as a P1 priority.
I'm sure you're not in ops. Or in a dev org of a service with decent request rates.
What you're asking for is a service to fail silently. There's no way a service with a decent request rate to have 0 500s. Not when it still sees development.
A 50 year old bank API? Maybe...
Is it more so to have something to link to for managers who aren't using the service have a pretty bar to look at and feel like they are "doing something"? Or is it more of a kind of a way to prevent confirming what you already suspect to be true. E.g. "Huh. Me and Jim are seeing problems. How about you Tom? Oh wait, crud. The service page is confirming it's down now. Never mind! Who wants coffee?!"
No, it's not. Official updates = potential SLA penalties. Always requires approval.
There's a threshold. It shows only once 1000 users complain.
/i
Can you sue companies for inducing such anxiety?
but I suppose that there might be some terms of conditions within using github (ahem Microsoft) that you can probably not sue them for something like this.
It really depends upon the severity of situation (imo)
For example, if a person had any heart condition and they got so stressed because of an error at github (which to be fair, I can understand the stress part, imagine losing some part of your software because it was on github and the amount of direct damage to livelihood if your income depended on it)
and I think that the judge might have to be in just the right technical know-spot as well and someone who can understand the situation from programmer's perspective hopefully.
Then I can see a case being made.
once again not a lawyer but an interesting question, would love reading other replies to your comment.
also for what its worth, you can sue any company for X,Y or Z. The question worth asking is if you can win such lawsuit.
Personally I believe it might be hard but not impossible but for all practical use cases it might as well be but the only answer can probably be found in court. I am just guessing at this point.
https://news.ycombinator.com/item?id=47237377
I vibe coded a script that interacts with both Gitlab and Github via their APIs and I've been using it pretty heavily since this morning. I crossed the streams! Goodness, I didn't know it would be _this_ bad!
spooky action at a distance
- So many super-heroes/super-villains
... You're off the hook this time./s
We can't be blocked here. Seems silly what we settled on this, but for a long time GitHub had been reliable enough for many years, but things are sliding down the pan as of late.
Been burned too many times on that one.
Move to EC2.
Darn AWS is down.
Alright, run it on a Mac Mini in your basement. Ahh dawn, your ISP is having issues. Good thing you have a backup 5G hotspot.
Ohh no, the power is out.
Eventually you have to trust someone else.
GitHub is a tragedy of the Commons. Too many people are using it, and Microsoft isn't willing to handle it correctly.
Feels like a very good business opportunity. Minimum 50k yearly contracts, GitHub with actual uptime. GitPro ?
Aggregate risk is too high.
This is supposed to be Hacker News! Who is coming up with a startup to fill the gap !
You should never entirely depend on a third party service to run your tests, either.
On my repo the jobs do not get scheduled on the PRs at all, so I assume that separation wouldn't help for todays issue.
Wait until you charge you for self-hosting runners.
Oh wait. They already tried.
You can now hire me as an overpriced consultant instead of paying Microsoft.
The latest language models have enabled this sort of thing for me. I can integrate a mini Jenkins into every project within a 5-10 minute prompting session. This sort of code isn't hard. It's just tedious, and the LLMs absolutely rock at boring repetitive stuff. Having a win32 service start up successfully on the very first try is something I haven't experienced until 2026.
I agree in a hosted+shared SQL scenario you have to be a little bit more careful with all of this. Arguably, you should have a separate schema management phase in these cases.
But if you are just SQLite embedded in the service, you can use the user_version pragma to track schema version and perform deterministic migrations (assuming a user didn't manually jack with the file in-between).
"Update something in the cloud" <- What do you mean?
That only works on extremely simple setups and has risks. If you have only a single server, you can stall it. Now, how to roll back?
https://www.reddit.com/r/GithubCopilot/comments/1toa9tf/mode...
[1] https://github.com/ericc-ch/copilot-api
So why are Actions so unreliable anyway? Occam's Razor would probably suggest the domain is inherently complex/difficult; but other providers show that reliability is possible. What would Occam's Razor suggest next? Poor management..?
You’d need at least some hash of sources + test results, and check that it matches that (in CI).
And you’d still deal with environment differences.
Reasonable concern. In ~10 years of indy development, I haven't forgotten to run tests before pushing to main, ever. So setting up and maintaining complicated machinery to solve a problem that could (but never has) happened doesn't justify taking focus off other more important things, namely building.
The benefit probably increases with team size (I'm a team of 1, so I appreciate the luxury of being able to dodge CI/CD entirely).
Say a disaster happens and someone pushes to main without running tests, 9 times out of 10 it will be of ~zero consequence (either the code works first time, it was a cosmetic change that hardly affected users etc).
I know there are horror stories and CI/CD would have prevented some of those, but IME they're just not that common nor severe for small operations, and even when they happen, only a small subset are irreversible/unfixable.
Basically, what you are suggesting is that everyone advertises their tests/builds run on slack? Also when two devs merge their changes, who compile/tests the master branch?
For small teams it could be as simple as everyone agreeing to ensure tests pass on main before pushing to prod.
Anyway. Forgejo's response to it: https://floss.social/@forgejo/116494295922963052
(Ofc, in a sensible universe, we just brush that off to a JS/Firefox glitch or my ISP.)
And yet, here I am. My code is not compiling, my AI isn't vibing, nonetheless I can't work! Two more hours before I can get off!
For Git, all you technically need is ssh access and some backup strategy for your server. It would be bare bones but workable. And there are of course plenty of OSS things that are a lot nicer than that.
I'm still using gh and gh actions and we are mostly below the freemium layer with that. But it is kind of slow and honestly a dedicated vm plus some high CPU/memory workers we can spin up on a need to have basis might be a lot faster. With GH outages becoming more common, my hand might be forced a bit.
In recent weeks, I've spun up listmonk (mailing list solution), matrix (as a slack alternative), and a few other things specific to our software stack. A github alternative would be more of the same. We don't need a lot.
The main objection is that with more moving parts to worry about, the workload for me also increases. Things need updating, monitoring, backups, alerting (and responding to alerts), etc. That sucks up my time and that is scarce.
Another reason for self hosting these days is that with agentic AI tools, self hosted things are a lot easier to integrate into agentic systems. If it is self hosted, you don't have to worry about API limitations, rate limitations, walled gardens, etc. All the traditional SAAS silos are becoming a problem from that point of view. The more locked down it is, the bigger the motive for moving away from it. That's why we ditched Slack for Matrix. Slack is hopelessly locked down and tedious to deal with. Matrix is super easy for this.
Technically Dropbox is just rsync.
Also https://xkcd.com/1319/ but for maintenance.
I don't think vibecoding at Github has much to do with it.
That makes sense. Thank you!
I don’t buy the excuse. I want to hitch my wagon to those “mysteriously lucky” competitors. (And have. And haven’t had similar issues to Github, since.)
Tough to say as this is all speculative, though.
Think critically.
agentic "ai" is going great
That being said there was a noticeable trend starting around 2022.[2] That being said they’ve also been doing a big migration to Azure. It’s likely a combination of things.
1: https://www.cnbc.com/2025/04/29/satya-nadella-says-as-much-a...
2: https://www.reddit.com/r/sysadmin/s/LOMPaSv3wY
https://damrnelson.github.io/github-historical-uptime/
https://news.ycombinator.com/item?id=47591928
When I dug in to the latest outages, they were almost all in small newer, features like all the AI stuff. The actual core GitHub platform seems much more stable than the unofficial uptime trackers propose.
There’s also this blurb from a story on HN the other day that supports what I’m seeing.
“ I've felt this way for a long time, but for the past month I've kept a journal where I put an "X" next to every date where a GitHub outage has negatively impacted my ability to work2. Almost every day has an X. On the day I am writing this post, I've been unable to do any PR review for ~2 hours because there is a GitHub Actions outage3. This is no longer a place for serious work if it just blocks you out for hours per day, every day.”
No, it's not like "act," because it uses the standard Github runner, the difference is that the control plane is an emulation of api.github.com, because of this we can do all kinds of nice things:
Caching in ~0 ms. Pause on failure, so you can let your AI agent fix it and retry without pushing.
Is what it boils down to.
> codex "Fix this pipeline, use `act` to verify your changes"
I have tried to use act many times, and many times I've failed.
P.S. pause on failure is also helpful for humans, but I'm trying to be realistic about where the future of programming is going...
I like that it exists, but what a freaking mess that it's necessary and so difficult to do.
I started playing with proxmox VMs and containers in them (docker and tart) to see if I can build some local infrastructure to properly solve this…
The jobs runs via containers.
For instance, the UI at setups such as https://git.devuan.org/Daemonratte/gtk2-ng is quite ok-ish, in my opinion. Granted, it is mostly copy/paste from github but that still is about 1000000x better than sourceforge's interface - and gitlab's UI too (I just hate gitlab's UI, they seem to love complexity and a billion features only 0.000001% ever need; GitHub, with all its faults, is for the most part really simple - not everywhere, e. g. GitHub wiki setup sucks, but by and large I think it is simple overall).
source: voices in my head. Not affiliated with MSFT.. anymore.
We're now considering Buildkite (apparently they have a GH actions migration tool) or self hosting something (GitLab CI, maybe even Jenkins), as it looks like that would've kept ticking over since we're still seeing webhooks being triggered today during the downtime.
[0] https://depot.dev
I used to use Cirrus CI as an alternative to GitHub Actions and am looking for a new alternative. I wonder if Depot could fit in the same way for my needs. I need to run builds and tests in Windows, Linux and macOS.
Hope you don't mind the public ask, it seems useful for others.
If we're using depot runners, and want to use them directly, or move off of github actions being the controller for when things run: what do you suggest?
Trigger the workflows directly on depot via CLI?
We’d need more details around what you’re seeing. It is true that if auth across GitHub is broken than we can’t copy your actions out to be used by Depot CI. However, we have a solution in the works for that as well.
In short, Depot CI, our own engine and control plane is not dependent on upstream actions control plane. But still has to listen for commit events to know if/when to run jobs on things like PRs. This to is being removed in the future.
[0] https://depot.dev/products/ci
https://www.blacksmith.sh/ and https://runs-on.com/
They also say that they're much cheaper than github
https://www.githubstatus.com/uptime?page=31
Is this your argument?
It is relatively easy to scale a collection of simple things to extreme and exhibit complex behavior together. It is a lot harder to scale something complex to extreme. But too many times the latter is the default - designed wrong from the ground up and stuck in scaling hell.
If Google owned GitHub would they be better positioned to scale?
Jesus, that's both horrible and seems within reach.
https://mrshu.github.io/github-statuses/
The external page linked above goes the other extreme and considers it a bad status whenever any individual service is degraded.
In reality the majority of people only use 3 or 4 of the core services the majority of the time but since there's no "core services" SLA/uptime the usability of github for the majority of people is slightly obfuscated.
I much prefer Woodpecker CI, which is an open source fork of Drone.io. It supports multiple Git backends like GitHub, Gitea, Forgejo, Gitlab, Bitbucket. It supports running jobs locally, on Docker, and on Kubernetes. And there's autoscalers built in for AWS, Hetzner, Linode, Vultr, and Scaleway. There's a bunch of 3rd party plugins (https://woodpecker-ci.org/plugins) for custom integrations. The UX is also very simple, with OAuth used not only for authentication/authorization but also setting up & accessing repos. The system architecture is great, with separate components that run stateless connected to a database, and a custom plugin is any program that takes environment variables and does stdio. The config file is a good balance of ugly YAML and convenience syntax like shell-style parameter expansion variables.
It probably takes less than 15 minutes to install, set up, and run WoodpeckerCI for a small team, so it's not a big investment to try out or host. With the autoscaling plugins it lets you scale your workload up to whatever size. Honestly you could run it on a laptop since it's written Go.
(to clarify for beginners: the config file docs are found in a section called "workflow syntax" (https://woodpecker-ci.org/docs/usage/workflow-syntax) and variable parameter expansion is buried deep in an environment variables page called "string operations" (https://woodpecker-ci.org/docs/usage/environment#string-oper...). poorly organized docs aside, the system itself works well)
From their FAQs[0]:
> Codeberg's mission is to promote free/libre software. Keeping software private is obviously not our primary use case, but we acknowledge that private repositories are useful or necessary at times.
0. https://docs.codeberg.org/getting-started/faq/
Perhaps not 100% physically shared infra but there's references of architecture overlap such as "The GitHub-hosted runner application is a fork of the Azure Pipelines Agent."
https://docs.github.com/en/actions/concepts/runners/github-h...
https://github.com/actions/runner-images#about
A few threads where blips have affected both services.
https://news.ycombinator.com/item?id=42781922
https://news.ycombinator.com/item?id=41001040
https://news.ycombinator.com/item?id=35003741
Setting it all up would have been tediously annoying eight months ago (Buildkite requires setting up GitHub webhooks for each repo).
Last week I just had codex set up everything, ephemeral vm runners and all, using a couple of low-spec refurb mac minis, Buildkite’s API, a short-lived API token, and migrate my repositories one by one.
So far so good, it’ll pay for itself within two to three months, and following today’s outage I suggested at work that we experiment with the same set up.
They’re considering it.
GitHub was, once upon a time, quite stable. Things have changed: more features, more usage, and automated agents.
"Microsoft’s GitHub was positioned to win the AI coding race. Outages got in the way" - https://www.cnbc.com/2026/05/22/microsoft-was-positioned-to-...
Even though it's selfhosted and we don't have a dedicated infrastructure team, I don't remember it ever being down in the last 12 years I have been working here.
Something’s wrong when my own infrastructure is more reliable than Microsoft’s.
EDIT: sorry i meant this rant at the one complaining for the free service not for the paid customers (which is unacceptable)
https://github.com/pricing
Technically this one was earlier but the other one has more traction.
[0] https://news.ycombinator.com/item?id=48278374
Reminds me of the occasional “JavaScript developer tries to vibe debug a Linux kernel issue” comments we get here.
"Well. It's got a nine in it"
"What percentage??"
"Nine"
Thanks for pointing out that nobody is using that thing
- GitHub
- Hiring budgets
- RAM (/personal computing in general)
- Electricity
- Media/Content
- Truth
I like being able to vote with my (teams) wallet and I'm tired of staying out of convenience
The open source contribution model as we once knew it is dead; you're not going to accept patches from random agents. The risk is way too high. And you can see that increasingly "AI Slop" makes it difficult to be a maintainer of any semblance of a popular repo.
So what's the value? A durable place to store work? hah.
Discovery? That part of Github has always been shitty.
So that leaves.. Github Actions? The thing that is down every other day and has been the subject of a few ~rug pulls~/attempted price hikes that are almost surely coming back?
This is a conservative estimate assuming linear growth, the actual number is likely going to be higher. Much higher.
It's not too hard to grow 14X YoY if you start from a hundred customers. If you have hundreds of millions? Yeah, not so easy.
[1] https://x.com/kdaigle/status/2040164759836778878
Or maybe it's before the GitHub internal devs are online and deploying changes.
We have already seen this in the last some weeks, but now this has become a meme that keeps on giving. GitHub down! GitHub up again. GitHub Down! GitHub ... ...
Perfect timing that we post https://www.jxd.dev/writing/building-plain just as this latest incident started.
I've done some hacky shit in CI scripts, but none made me more mad than that one.
With all the recent negativity – how are they not even TRYING to fix the damn thing?
Self hosted Gitlab with self hosted (or AWS) runners running your pipelines.. We only use Github as a mirror for our public repositories.
I am trying to refrain my "off topic" rants... but such microsoft github abuse is generating so much hate due to their dominant market position, it is hard.
This is why we don't use Github Actions, kids.
Seriously, its a proprietary build service that puts the keys to the kingdom in someone elses' control. Just: No!
Print this status page to PDF so you've got it handy next time someone castigates you for not using Github Actions, folks.
This time today it was caused by friendly fire by the automatic suspension of the GitHub Actions bot which is now a "Ghost" user. Since there is no CEO of GitHub to contact it we are just going to see more [1] of this again.
You might need to push a critical change soon, but now you cannot. You won't get any of these issues if you self hosted as I said 6 years ago...[2]
[0] https://www.githubstatus.com/incidents/g6ffrm0rfvz9
[1] https://news.ycombinator.com/item?id=48085501
[2] https://news.ycombinator.com/item?id=22867803
I don't want to delve into it any further - but something about it seems incongruous. It's not spam it's submarine marketing.
Apologies for the spam!
I'm guessing related to this? The blog post is dated 11 days ago but I just noticed a blue banner on my actions page today.