Lasting memory programming
June 2013 I wrote about the future of the interface nonvolatile memory (NVM) in. Which describes the SNIA NVM Programming technical work group (TWG ) NVM programming model being developed. In the past four years, published specifications, as predicted, the programming model has become the focus of a lot of follow-up. The programming model, as described in the specification NVM.PM.FILE, PM may be as the operating system files are mapped into memory. This article describes how to achieve lasting memory programming model in the operating system, what work has been done, and what challenges we have faced.
BACKGROUND persistent memory
PM and the same storage class memory term, has byte addressing, a load / store memory access characteristics, but with persistence. This article concerns the PM hung on the system memory bus, e.g. DRAM DIMM, to create a class of non-DIMMs called NVDIMMs.
What To further illustrate the lasting memory said that the discussion only NVDIMMs, like software allows access to the same memory access. Providing all the advantages of memory semantics, e.g. CPU CACHE consistency, the DMA direct memory access other devices, to access a buffer line size, byte addressing can. To provide these semantics, the delay devices when the CPU must be sufficiently reasonable to block access to CPU instructions. When the slower persistence as NAND flash memory, because of the need to access in units of blocks, and requires a long enough time to context switch. Hardware access time is typically in milliseconds, NAND Flash SSD with microsecond units, PM in nanoseconds. Dependent on the hardware media type, NVDIMM may speed less than the DRAM, but his speed is comparable to the speed.
Now some NVDIMM products on the market, use runtime DRAM as a medium, the content is automatically backed up to the NAND Flash during power down, power up again when the NAND Flash content to return DRAM. These products provide DRAM performance, but requires additional components and batteries to store the data, as compared to DRAM, for each of the DIMM to provide a small-capacity, high consumption characteristics of each G. Non-volatile media emerging, such as 2015, Intel and Micron joint research and development of 3D XPoint technology, through higher capacity than DRAM. T reached on each CPU bandwidth, the lasting memory frontier attracted much attention: durability, capacity and consumption.
Lasting memory programming model
of how to access the application is lasting memory? And a volatile memory different applications require specific methods and content specified lasting connection; lasting memory not as volatile memory is anonymous, he needs to file the same name as a region, so that the application can find him. Applications require access to persistent memory control authority. SNIA TWG recommended programming model because the operating system can use standard file naming semantics provide a lasting memory, permissions, and memory mapping.
Currently, a variety of operating systems including Linux and Windows are supported by this model.
DAX
lasting memory file system adapter shown in Figure 1, allows direct access to persistent memory, without going through the page cache system. Such a characteristic is called DAX. Lasting memory programming model and DAX properties suggest a lasting memory file can use mmap () or MapViewOfFile () function is similar signs mapped into memory. 1 forms the rightmost view of this type. Applications direct access to persistent memory by load / store instructions. Persistence allows direct access to the media without switching to kernel mode and user mode.
Persistent storage
Linux systems can use msync () or fsync () to ensure data persistence, Windows can ensure data persistence through FlushViewOfFile () and FlushFileBuffers (). These calls will create a memory gate, data prior to this point have all been persistent to persistent memory. Historically, the memory gate operating system needs to find dirty pages in the page cache, and then they flushed to disk. Due to persistent memory without using the page cache, the operating system will only need to change CPU cache is flushed to the lasting memory. 2: 
in dotted line in Figure 2 shows the lasting domain. This level architecture, data in the dotted line portion of either the DIMM, or how the write request queue WPQ memory controller. Either return, lasting memory needs to have enough power to data dashed box flushed to persistent media. This feature is called asynchronous DRAM brush, already have this characteristic and NVDIMM.
X86 architecture, simply executing instructions stored does not ensure data persistence, because the data may still be in the CPU cache, once the power failure, the data will be lost. The need for additional instruction flashed ensure data persistence. The following table describes how they work.
Figure 2 and Table 1 may be confusing, why Intel CPU cache will not get lasting domain part. Technically feasible, including CPU CACHEs 2 within the dashed box in FIG.
Persistent domain extensions including x86 CPU CACHEs problem is the very large caches, he needs much more power than the actual capacitor to provide power. This means that the platform needs batteries. At this time, the server supports persistent memory with a battery is not realistic. But for hardware vendors, of course, it may contain a battery in its commodities. This allows you to skip described in Table 1 read buffer refresh command, but sfence instruction is still necessary because only stored in the global memory barrier visible only to be considered persistent, and this is sfence ensured.
Because the application vendor plans to use the battery as well as future expectations on all platforms include the CPU cache to persistent domain, so add a property in ACPI, so that when the brush skip CPU, BIOS may notify the operating system. This allows the operating system in an optimal manner similar msync call.
Flashing a persistent user space domain
WBINVD exception, Intel CPU user mode instruction modes support Table 1 below. Use CLWB (CLFLUSHOT or CLFLUSH) Write cache line brush with the user and supports the use of temporary storage mode. 
This allows the user space flushed to persistent memory, without going through the kernel, this feature called Optimized Flush. It depends on the operating system and hardware platform for each selective support this feature. Despite CPU support, but only for applications, operating system, said when the safe use of optimized flush. When the file system metadata changes require msync flashed, the operating system needs this control point.
The current implementation supports secure user space brush is constantly evolve. The DAX is provided by the Windows NTFS file system, including the unconditional support Optimized flush. Windows uses the command similar CLWB + SFENCE data persistence to maintain a lasting memory. Linux is ext4 and xfs support DAX, regardless of the user space flashed security. As a temporary solution, Linux provides Device-DAx, allows an application to open a lasting memory device, map it into memory, use a brush to write the user space to ensure durability.
Libpmem library provides function tells the application when Optimized flush is safe. Programmers are strongly encouraged to use libpmem to identify and use a brush to write the user space. Libpmem is also used in the case of using a battery detection platform, writing brush sfence simply calls into call instructions. This library will be described in detail below.
Lasting memory challenge
when the in-memory data structure change atomicity problem arises. It will test the changes to the data only when half of the other threads access the data structure? Usually to protect the data structure of a lock when multi-threaded programming. It is also sometimes used to ensure atomic instruction in hardware. This paper has also become the atom visibility, when the changes are committed, to see another thread to modify this thread. 
Libpmemobj library provides transactional guarantees to ensure the safety off. In lasting memory before the power fails interrupted when writing, memory status no problem, because it is volatile. However, persistent memory, to be understood that part of the status has persisted after the brush. Intel only 8 bytes of storage used to ensure atomicity failure. Greater than 8 bytes can not guarantee data consistency.
Other challenges: management space. Because the field is a lasting memory as a file, the file system can manage this space, but once the application is mapped into memory, what happens depends entirely on the application file. And similar malloc function allocates memory is volatile, when the restart method does not provide a lasting memory reconnection right, do not quit any steps to ensure data consistency when a fault occurs. So lasting memory programming also requires focusing on space allocation issues.
Address Independence is another challenge. Although it is technically possible to achieve lasting inner mapped to the same address, but when other items to change the size of the map, which is unrealistic. An address space layout randomization feature will adjust the operating system libraries, and random file mapping address. Independent means that address persistent in-memory data structure refers to another data structure using a pointer, even if the file is mapped to a different address, the pointer must be used in some way. There are several ways to achieve this, for example, reposition the pointer after the mapping, using the pointer relative rather than absolute pointer, or using some type of object ID to the application data structures reside in persistent memory.
NVM library
Intel developed pmdk library on GitHub open source, open source license BSD, manual can be viewed from http://pmem.io.
libpmem: Basic Library
This library is relatively small, relatively simple, which includes detection CPU instructions and use a brush to write the best copy instruction range support.
libpmemobj: support services
Libpmemblk and libpmemlog: support a specific user case
Libmemkind: persistent volatile memory using a
summary of
2013's ideas matured and added a complete programming model. Pmdk library was developed to provide a lasting memory programming. The library on GitHub open source https://github.com/pmem
original
https://www.snia.org/pm-summit2019
https://www.snia.org/pm-summit2017