Hosting incident: coordinate recovery and write a useful support request
When a service fails, the first few actions can either clarify the incident or erase the information needed to understand it. A useful response separates what users observe, what the team knows and what remains a hypothesis. Keep one coordinated timeline and give each change an owner, a reason and a way to check the result.
The practical takeaway
Stabilise the situation without losing useful evidence, share a concise factual timeline, make controlled changes, and verify recovery through the affected customer journey.
Establish the impact and give coordination an owner
Begin with what has stopped working and who is affected. A public website outage, delayed background jobs and an unavailable administration screen have different consequences. Note the first known failure time, the timezone, the affected service or domain and whether the issue is continuous or intermittent. Keep those observations separate from assumptions such as “the server is down” when only one application endpoint has been checked.
Assign someone to coordinate the response and maintain the timeline, even if the same person also performs technical work in a small team. Identify who can approve a risky change and who communicates with affected people. Several administrators making independent changes can remove the ability to tell which action helped. Use one shared incident record and a clear handover when the person on duty changes.
Check whether the incident has a security dimension, such as an unexpected account change, altered content or exposed credentials. A suspected compromise needs a response that considers containment and evidence, not only a quick restart. Keep any contractual notification or legal obligations with the appropriate responsible people. This operational guide does not decide those obligations, and a normal support ticket is not a substitute for an organisation’s security response process.
Preserve a useful record before changing the system
Record the current symptoms and relevant recent changes before they disappear from memory. Include deployment times, configuration edits, certificate or DNS changes and scheduled tasks that overlap the failure. Collect narrowly relevant logs using the available tools and permissions. A systemd journal can be filtered by time and service; application logs may have their own correlation identifiers. Keep a note of where each extract came from and the clock context used.
Limit the data you collect and share to what the investigation needs. Logs can contain session identifiers, request parameters, personal data or secrets accidentally written by an application. Redact the copy sent to support while preserving the original privately when appropriate. Do not paste an entire environment file or a database dump into a ticket. A public transaction reference can be useful for a payment investigation; a wallet private key never is.
| Keep | Why it helps | Avoid sharing |
|---|---|---|
| Service or order reference | Identifies the owned resource | Passwords and recovery keys |
| Time range with timezone | Aligns observations and logs | An unexplained time from an unknown clock |
| Exact error and affected journey | Makes the symptom reproducible | An unfiltered export containing customer data |
| Recent actions and their results | Prevents repeated or conflicting changes | Speculation presented as confirmed cause |
Choose the smallest intervention that fits the evidence
State the hypothesis and the expected result before a change. If a deployment introduced the error, a prepared application rollback may be more informative than changing unrelated network settings. If the filesystem is full, deleting arbitrary files can destroy useful data while leaving the cause untouched. Determine what can be removed or moved safely, preserve the relevant record and check the application’s recovery behaviour after space becomes available.
A restart can be a reasonable recovery action in the right circumstances, but it also changes the state you are investigating. Consider active writes, queued work and the information that will be lost. When an immediate restart is necessary to restore service, record the decision and the result. Repeated restarts without a new hypothesis can turn an intermittent fault into a long sequence of missing evidence.
For suspected compromise, involve the responsible security lead and use an authorised containment plan. Restricting exposure, isolating a workload, preserving relevant evidence and replacing affected credentials may have different priorities depending on the situation. Do not experiment with an attacker’s files or tools on another production system. Keep investigation copies and access tightly controlled, and avoid allowing an uncertain recovery action to spread the incident into an otherwise unaffected environment.
Give support a brief that advances the investigation
Open a request from the account that owns the affected service when possible. Start with the impact and the reference, then provide the observed time range, exact error, reproducible journey and relevant recent changes. Mention the actions already attempted and whether each improved, worsened or did not change the symptom. If the issue is intermittent, include a small number of representative timestamps instead of saying that it fails randomly.
- Affected service, domain or owned order reference.
- Observed start time and timezone, plus representative failing times.
- Expected behaviour and the exact result a user sees.
- Safe, relevant log extracts or public transaction references.
- Recent changes and attempted recovery actions with their outcomes.
- The next decision or infrastructure fact you need the team to clarify.
A useful example is a report that an application’s login returns a specific error after a named deployment, while its public home page still works, with three failing timestamps and a redacted application trace. That gives the next person a concrete starting point. “Everything is broken, please fix it” does not. Keep follow-up observations in the same conversation so the team can see the incident’s development rather than reconstructing several disconnected requests.
Verify recovery and turn the incident into a maintenance improvement
Check the journey that originally failed, not just whether a process is running. Confirm that pending work is progressing, data is coherent and authorised users can perform the important actions. Continue observation long enough to catch the recurrence pattern relevant to the incident. If a temporary exception or workaround was used, record its owner and removal plan. A quiet error log is not enough if the application is silently dropping work.
Close the response with a concise timeline, confirmed cause where established, contributing conditions and practical follow-up. Distinguish a confirmed cause from an explanation that remains plausible but unproven. Assign actions such as testing a restore, adjusting deployment checks or improving a missing diagnostic. The purpose is to make the next incident easier to detect and handle, with a record that another maintainer can understand without having been present.
