
Ceph
Distributed storage for environments that require High Availability, scalability and fault tolerance. We use Ceph in Proxmox clusters, Private Cloud architectures and other infrastructure where storage must remain available despite hardware failures.
Distributed Storage • High Availability • Private Cloud
Ceph is an Open Source distributed storage platform designed for building scalable and fault-tolerant data storage environments. It can provide block storage for virtual machines, a shared file system and object storage compatible with the S3 API.
At sLogic Group, we use Ceph primarily as a storage layer for Proxmox VE clusters, Private Cloud environments and infrastructure that requires High Availability.
Ceph makes it possible to move away from a single storage array towards a distributed architecture in which data can survive the failure of a disk, server or another defined infrastructure component.
One cluster, multiple storage models
Ceph uses a distributed RADOS layer on top of which different data-access models can operate. This allows the same technology to serve several different roles within the infrastructure.
RBD — Block Storage
Distributed block storage well suited to virtual machines and KVM environments. In Proxmox VE it can provide shared storage for the entire cluster.
CephFS — File Storage
A distributed POSIX file system that enables data sharing between multiple systems and applications.
RGW — Object Storage
An object storage layer exposing an S3-compatible API for applications and systems that use object storage.
Ceph as storage for Proxmox VE
One of the most common uses of Ceph is building shared storage for a Proxmox VE cluster.
Proxmox integrates directly with Ceph and allows its components to run on the same physical nodes that host virtual machines and containers. This creates a hyperconverged infrastructure combining compute and storage layers.
Virtual machines are no longer dependent on the local disk of a single host. Their data is stored across the distributed cluster and can remain available to other Proxmox nodes.
Compute
Proxmox nodes provide the CPU and memory required to run virtual machines and containers.
Storage
Disks installed in the nodes form a shared distributed Ceph storage layer available to the entire cluster.
High Availability
A properly designed cluster reduces dependency on a single host or a single storage device.
Proxmox provides the virtualisation layer. Ceph can provide distributed storage. Only a properly designed combination of both layers creates a resilient Private Cloud infrastructure.
How does Ceph work?
Ceph does not rely on a single central storage array or one controller responsible for all data. The cluster consists of multiple cooperating components.
OSD
Components responsible for storing data on storage devices and performing operations related to replication and data recovery.
MON
Monitors maintain information about cluster state and topology. For availability, they are deployed using a properly designed quorum.
MGR
Ceph Manager provides data used for monitoring, management and additional cluster modules.
CephFS also uses MDS servers responsible for handling file-system metadata.
CRUSH and data placement
One of the key elements of Ceph architecture is the CRUSH algorithm. It determines how data is distributed across the cluster without depending on a single central location table.
The architecture can account for different failure domains, for example:
- disk,
- server,
- rack,
- room or infrastructure zone.
With properly designed CRUSH rules, replicas can be placed so that the failure of a defined infrastructure component does not remove every copy of the data.
Redundancy only provides value when copies of the data are located on different sides of the failure scenario we want to protect against.
Replication or Erasure Coding?
Ceph supports different data-protection mechanisms. The choice should depend on workload type, available capacity, expected performance and required resilience.
Replicated Pool
Data is stored in multiple copies. This is a straightforward model often selected for virtual-machine storage where predictable performance and fast recovery are important.
Erasure Coding
Data is divided into data and parity fragments. It can reduce capacity overhead, but introduces different performance and architectural requirements.
Designed for the workload
There is no single correct profile for every use case. Storage for virtual machines is designed differently from archive or object storage.
Ceph requires the right network
Distributed storage can generate significant network traffic. Data moves between clients and OSDs, while failure recovery or disk replacement also causes additional recovery and rebalancing traffic.
A Ceph design should therefore consider:
- network interface bandwidth,
- link redundancy,
- switch performance,
- latency,
- traffic segmentation,
- expected recovery and rebalancing traffic.
Fast NVMe drives alone will not provide the expected performance if the network between nodes becomes the bottleneck.
Ceph performance depends on the entire architecture
Ceph performance cannot be evaluated only by looking at the type of disks installed in the servers.
Important factors include:
- number of hosts and OSDs,
- type of HDD, SSD or NVMe devices,
- network performance,
- replication or Erasure Coding profile,
- workload type and I/O size,
- number of concurrent clients,
- available free capacity,
- cluster behaviour during recovery.
A cluster should be designed not only for performance during normal operation, but also for how the infrastructure behaves during a failure and data recovery.
Scaling without replacing the entire storage array
One of the advantages of distributed architecture is the ability to expand the cluster by adding more resources.
Depending on the design, it may be possible to increase:
- the number of disks,
- node capacity,
- the number of storage servers,
- network performance,
- compute resources in a hyperconverged environment.
After the topology changes, Ceph can redistribute data across available OSDs. This does not mean that every cluster can be expanded without limits — future growth should be considered during the initial architecture design.
Ceph does not replace backup
Data distribution and replication primarily protect against infrastructure failures. They do not provide historical recovery points required after accidental deletion, application errors or ransomware attacks.
If a virtual machine is deleted or its data is encrypted, Ceph can correctly replicate that new state across the required OSDs.
Ceph improves storage availability. Backup allows data to be recovered from an earlier point in time. One does not replace the other.
A Ceph environment should therefore be protected by an independent Backup and Disaster Recovery system.
When does Ceph make sense?
Ceph is particularly worth considering when an organisation:
- is building a multi-node Proxmox VE cluster,
- needs shared storage without relying on a single central storage array,
- is designing a Private Cloud or hyperconverged infrastructure,
- needs storage that can scale as the environment grows,
- wants to reduce dependency on a proprietary storage platform from a single vendor,
- has requirements for high data availability.
When might Ceph not be the best choice?
Ceph should not be deployed simply because it is distributed and Open Source.
In a small environment with only one or two servers, its complexity may outweigh the benefits. Similarly, where storage requirements are limited, a simpler architecture based on ZFS or a dedicated storage array may be more appropriate.
Before choosing Ceph, it is worth assessing:
- environment scale,
- availability requirements,
- I/O profile,
- required capacity,
- network infrastructure,
- budget,
- skills required for ongoing operations.
Ceph is a tool for solving a specific infrastructure problem — not a mandatory component of every Proxmox cluster.
Monitoring a Ceph cluster
Distributed storage should be continuously monitored. A simple HEALTH_OK status should not be the only metric being observed.
Monitoring should also cover areas such as:
- OSD, MON and MGR status,
- Placement Group state,
- capacity utilisation across individual OSDs,
- storage-operation latency,
- disk performance and errors,
- network state between nodes,
- recovery and rebalancing operations,
- uneven storage utilisation,
- growth rate of used capacity.
Ceph can be integrated with external monitoring systems, allowing storage issues to be correlated with the state of hosts, networking and virtual machines.
How do we design Ceph environments?
01. Requirements
We define capacity, performance, availability, workload and future growth requirements.
02. Architecture
We select the number of nodes, storage devices, network architecture, failure domains, data pools and integration with the virtualisation layer.
03. Implementation and testing
We deploy the cluster, configure monitoring and test performance as well as behaviour during controlled failure scenarios.
Ceph as part of a Private Cloud
Ceph should not be designed in isolation from the other infrastructure layers. Storage needs to work together with virtualisation, networking, backup, monitoring and Disaster Recovery requirements.
For larger projects, we therefore treat Ceph as part of the overall Infrastructure and Private Cloud architecture rather than as a standalone system to install.
A good Ceph design starts with application and infrastructure requirements — not with the number of disks currently available in the servers.
How can we help?
sLogic Group designs and implements Ceph environments for Proxmox VE clusters, Private Cloud and infrastructure that requires distributed storage.
The scope can include assessment of the existing infrastructure, cluster architecture, storage and network design, Ceph configuration, Proxmox VE integration, monitoring, performance testing and planning for future expansion.
If you are not yet sure whether Ceph is the right solution for your environment, we can start with an IT Audit and Architecture.
Designing a Proxmox cluster or Private Cloud?
We can assess your storage, performance and availability requirements and design a Ceph architecture that fits the real needs of your environment.