/*++ Copyright (c) 1991 Microsoft Corporation Copyright (c) 1991 Nokia Data Systems AB Module Name: dlcbuf.c Abstract: This module implements the DLC buffer pool manager and provides routines to lock and unlock transmit buffers outside of the buffer pool DLC has a buffering scheme inherited from its progenitors right back to the very first DOS implementation. We must share a buffer pool with an application: the application allocates memory using any method it desires and gives us a pointer to that memory and the length. We page-align the buffer and carve it into pages. Any non page-aligned buffer at the start or end of the buffer are discarded. Once DLC has a buffer pool defined, it allocates buffers in a fashion similar to binary-buddy, or in a method that I shall call 'binary spouse'. Blocks are initially all contained in page sized units. As smaller blocks are required, a larger block is repeatedly split into 2 until a i-block is generated where 2**i >= block size required. Unlike binary-buddy, the binary-spouse method does not coalesce buddy blocks to create larger buffers once split. Once divorced from each other, binary spouse blocks are unlikely to get back together. BufferPoolAllocate is the function that single-handedly implements the allocator mechanism. It basically handles 2 types of request in the same routine: the first request is from BUFFER.GET where a buffer will be returned to the app as a single buffer if we have a block available that is large enough to satisfy the request. If the request cannot be satisfied by a single block, we return a chain of smaller blocks. The second type of request is from the data receive DPC processing where we have to supply a single block to contain the data. Luckily, through the magic of MDLs, we can return several smaller blocks linked together by MDLs which masquerade as a single buffer. Additionally, we can create buffers larger than a single page in the same manner. This receive buffer must later be handed to the app in the same format as the buffer allocated by BUFFER.GET, so we need to be able to view this kind of buffer in 2 ways. This accounts for the complexity of the various headers and MDL descriptors which must be applied to the allocated blocks Contents: BufferPoolCreate BufferPoolExpand BufferPoolFreeExtraPages DeallocateBuffer AllocateBufferHeader BufferPoolAllocate BufferPoolDeallocate BufferPoolDeallocateList BufferPoolBuildXmitBuffers BufferPoolFreeXmitBuffers GetBufferHeader BufferPoolDereference BufferPoolReference ProbeVirtualBuffer AllocateProbeAndLockMdl BuildMappedPartialMdl UnlockAndFreeMdl Author: Antti Saarenheimo 12-Jul-1991 Environment: Kernel mode Revision History: --*/ #include #include #include "dlcdebug.h" // // LOCK/UNLOCK_BUFFER_POOL - acquires or releases per-buffer pool spin lock. // Use kernel spin locking calls. Assumes variables are called "pBufferPool" and // "irql" // #define LOCK_BUFFER_POOL() KeAcquireSpinLock(&pBufferPool->SpinLock, &irql) #define UNLOCK_BUFFER_POOL() KeReleaseSpinLock(&pBufferPool->SpinLock, irql) // // data // PDLC_BUFFER_POOL pBufferPools = NULL; #define CHECK_FREE_SEGMENT_COUNT(pBuffer) /* Enable this, if the free segment size checking fails: #define CHECK_FREE_SEGMENT_COUNT(pBuffer) CheckFreeSegmentCount(pBuffer) VOID CheckFreeSegmentCount( PDLC_BUFFER_HEADER pBuffer ); VOID CheckFreeSegmentCount( PDLC_BUFFER_HEADER pBuffer ) { PDLC_BUFFER_HEADER pTmp; UINT FreeSegments = 0; for (pTmp = (pBuffer)->FreeBuffer.pParent->Header.pNextChild; pTmp != NULL; pTmp = pTmp->FreeBuffer.pNextChild) { if (pTmp->FreeBuffer.BufferState == BUF_READY) { FreeSegments += pTmp->FreeBuffer.Size; } } if (FreeSegments != (pBuffer)->FreeBuffer.pParent->Header.FreeSegments) { DbgBreakPoint(); } } */ /*++ DLC Buffer Manager ------------------ The buffer pool consists of virtual memory blocks, that must be allocated by the application program. The buffer block descriptors are allocated separately from the non-paged pool, because they must be safe from any memory corruption done in by an application. The memory allocation strategy is binary buddy. All segments are exponents of 2 between 256 and 4096. There is no official connection with the system page size, but actually all segments are allocated within a page and thus we minimize MDL sizes needed for them. Contiguous buffer blocks decrease also the DMA overhead. The initial user buffer is first split to its maximal binary components. The components are split further in the run time, when the buffer manager runs out of the smaller segments (eg. it splits a 1024 segment to one 512 segment and two 256 segments, if it run out of 256 segments and there were no free 512 segments either). The clients of the buffer manager allocates buffer lists. They consists of minimal number of binary segments. For example, a 1600 bytes buffer request would return a list of 1024, 512 and 256 segments. The smallest segment is the first and the biggest is the last. The application program must deallocate all segments returned to it in the receive. The validity of all deallocated segments is checked with a reservation list. Buffer Manager provides api commands: - to initialize a buffer pool (pool constructor) - to add locked and mapped virtual memory to the buffer - to deallocate the buffer pool (destructor) - to allocate a segment list (allocator) - to deallocate a segment list (deallocator) - to set Thresholds for the minimum buffer size --*/ /*++ MEMORY COMMITMENT The commitement of buffer pools is a special service expecially for the local busy state management of the link stations. By default the uncommitted memory is the same as the free memory in the buffer pool minus the minimum free Threshold, but when a link enters to a busy state we know how much buffer space the link will need to receive at least the next frame. Actually we will commit all I- packets received in the local busy state. The local 'out of buffers' busy state will be cleared only when there is enough uncommited space in the buffer pool to receive all expected packets. We still indicate the local busy state to user, because the flow control function can expand the buffer pool, if it is necessary. We will just queue the clear local busy state command to a command queue (even if we complete it immediately), we don;t execute the queued command before there is enough uncommited space to enable the link receive. The buffer space is committed by the size of all expected packets, when the local busy state of a link is cleared. All received packets are subracted from the commited buffer space as far as the link has any committed memory. This may happen only after a local busy states. We will provide three macroes to BufGetPacketSize(PacketSize) - returns probable size of packet in buffers BufGetUncommittedSpace(hBufferPool) - gets the current uncommited space BufCommitBuffers(hBufferPool, BufferSize) - commits the given size BufUncommitBuffers(hBufferPool, PacketSize) - uncommites a packet --*/ NTSTATUS ProbeVirtualBuffer( IN PUCHAR pBuffer, IN LONG Length ); NTSTATUS BufferPoolCreate( #if DBG IN PDLC_FILE_CONTEXT pFileContext, #endif IN PVOID pUserBuffer, IN LONG MaxBufferSize, IN LONG MinFreeSizeThreshold, OUT HANDLE *pBufferPoolHandle, OUT PVOID* AlignedAddress, OUT PULONG AlignedSize ) /*++ Routine Description: This routine performs initialization of the NT DLC API buffer pool. It allocates the buffer descriptor and the initial header blocks. Arguments: pFileContext - pointer to DLC_FILE_CONTEXT structure pUserBuffer - virtual base address of the buffer MaxBufferSize - the maximum size of the buffer space MinFreeSizeThreshold - the minimum free space in the buffer pBufferPoolHandle - the parameter returns the handle of buffer pool, the same buffer pool may be shared by several open contexts of one or more dlc applications. AlignedAddress - we return the page-aligned buffer pool address AlignedSize - and the page-aligned buffer pool size Return Value: Returns NTSTATUS is a NT system call fails. --*/ { NTSTATUS status; PDLC_BUFFER_POOL pBufferPool; PVOID pAlignedBuffer; INT i; register PPACKET_POOL pHeaderPool; ASSUME_IRQL(DISPATCH_LEVEL); // // page-align the buffer // pAlignedBuffer = (PVOID)(((ULONG_PTR)pUserBuffer + PAGE_SIZE - 1) & -(LONG)PAGE_SIZE); // // and make the length an integral number of pages // MaxBufferSize = (MaxBufferSize - (ULONG)((ULONG_PTR)pAlignedBuffer - (ULONG_PTR)pUserBuffer)) & -(LONG)PAGE_SIZE; // // the buffer size must be at least one page (BufferSize will be 0 if // this is not so) // if (MaxBufferSize <= 0) { return DLC_STATUS_BUFFER_SIZE_EXCEEDED; } // // if MinFreeSizeThreshold < 0, we can have problems since it is negated later // on leaving buffer pool uninitialized // if (MinFreeSizeThreshold < 0) { return DLC_STATUS_INVALID_BUFFER_LENGTH; } // // if the size of the buffer is less than the minimum lock size then we lock // the entire buffer // if (MaxBufferSize < MinFreeSizeThreshold) { MinFreeSizeThreshold = MaxBufferSize; } // // allocate the DLC_BUFFER_POOL structure. This is followed by an array // of pointers to buffer headers describing the pages in the buffer pool // pBufferPool = ALLOCATE_ZEROMEMORY_DRIVER(sizeof(DLC_BUFFER_POOL) + sizeof(PVOID) * BYTES_TO_PAGES(MaxBufferSize) ); if (!pBufferPool) { return DLC_STATUS_NO_MEMORY; } // // pHeaderPool is a pool of DLC_BUFFER_HEADER structures - one of these // is used per page locked // pHeaderPool = CREATE_BUFFER_POOL_FILE(DlcBufferPoolObject, sizeof(DLC_BUFFER_HEADER), 8 ); if (!pHeaderPool) { FREE_MEMORY_DRIVER(pBufferPool); return DLC_STATUS_NO_MEMORY; } // // initialize the buffer pool structure // pBufferPool->hHeaderPool = pHeaderPool; KeInitializeSpinLock(&pBufferPool->SpinLock); // // UncommittedSpace is the space above the minimum free threshold in the // locked region of the buffer pool. We set it to the negative of the // minimum free threshold here to cause BufferPoolExpand to lock down // the number of pages required to commit the minimum free threshold // pBufferPool->UncommittedSpace = -MinFreeSizeThreshold; // // MaxBufferSize is the size of the buffer pool rounded down to an integral // number of pages // pBufferPool->MaxBufferSize = (ULONG)MaxBufferSize; // // BaseOffset is the page-aligned address of the buffer pool // pBufferPool->BaseOffset = pAlignedBuffer; // // MaxOffset is the last byte + 1 (?) in the buffer pool // pBufferPool->MaxOffset = (PUCHAR)pAlignedBuffer + MaxBufferSize; // // MaximumIndex is the number of pages that describe the buffer pool. This // number is irrespective of the locked state of the pages // pBufferPool->MaximumIndex = (ULONG)(MaxBufferSize / MAX_DLC_BUFFER_SEGMENT); // // Link all unlocked pages to a link list. // Put the last pages in the buffer to the end of the list. // for (i = (INT)pBufferPool->MaximumIndex - 1; i >= 0; i--) { pBufferPool->BufferHeaders[i] = pBufferPool->pUnlockedEntryList; pBufferPool->pUnlockedEntryList = (PDLC_BUFFER_HEADER)&pBufferPool->BufferHeaders[i]; } for (i = 0; i < DLC_BUFFER_SEGMENTS; i++) { InitializeListHead(&pBufferPool->FreeLists[i]); } InitializeListHead(&pBufferPool->PageHeaders); // // We can now lock the initial page buffers for the buffer pool. // The buffer pool allocation has been failed, if the procedure // returns an error. // #if DBG status = BufferPoolExpand(pFileContext, pBufferPool); #else status = BufferPoolExpand(pBufferPool); #endif if (status != STATUS_SUCCESS) { // // We must use the standard procedure for deallocation, // because the memory locking may have succeeded partially. // The derefence free all resources in the buffer pool. // BufferPoolDereference( #if DBG pFileContext, #endif &pBufferPool ); } else { KIRQL irql; // // Save the buffer pool handle to the link list of // buffer pools // ACQUIRE_DLC_LOCK(irql); pBufferPool->pNext = pBufferPools; pBufferPools = pBufferPool; RELEASE_DLC_LOCK(irql); *pBufferPoolHandle = pBufferPool; *AlignedAddress = pAlignedBuffer; *AlignedSize = MaxBufferSize; } return status; } NTSTATUS BufferPoolExpand( #if DBG IN PDLC_FILE_CONTEXT pFileContext, #endif IN PDLC_BUFFER_POOL pBufferPool ) /*++ Routine Description: The function checks the minimum and maximum size Thresholds and locks new pages or unlocks the extra pages and deallocates their buffer headers. The procedure uses the standard memory management functions to lock, probe and map the pages. The MDL buffer is split to smaller buffers (256, 512, ... 4096). The orginal buffer is split in the 4 kB even address (usually page border or even with any page size) to minimize PFNs associated with the MDLs (each MDL needs now only one PFN, to make DMA overhead smaller and to save locked memory). This procedure does not actually assume anything about the paging, but it should work quite well with any paging implementation. This procedure MUST be called only from the synchronous code path and all spinlocks unlocked, because of the page locking (the async code in always on the DPC level and you cannot make pagefaults on that level). Arguments: pBufferPool - handle of buffer pool data structure. Return Value: NTSTATUS Success - STATUS_SUCCESS Failure - DLC_STATUS_NO_MEMORY --*/ { NTSTATUS status = STATUS_SUCCESS; PDLC_BUFFER_HEADER pBuffer; KIRQL irql; ASSUME_IRQL(DISPATCH_LEVEL); LOCK_BUFFER_POOL(); // // UncommittedSpace < 0 just means that we've encroached past the minimum // free threshold and therefore we're in need of more buffer space (hence // this function) // if (((pBufferPool->UncommittedSpace < 0) || (pBufferPool->MissingSize > 0)) && (pBufferPool->BufferPoolSize < pBufferPool->MaxBufferSize)) { UINT FreeSlotIndex; while ((pBufferPool->UncommittedSpace < 0) || (pBufferPool->MissingSize > 0)) { pBuffer = NULL; // // if there are no more pages to lock or we can't allocate a header // to describe the buffer then quit // if (!pBufferPool->pUnlockedEntryList || !(pBuffer = ALLOCATE_PACKET_DLC_BUF(pBufferPool->hHeaderPool))) { status = DLC_STATUS_NO_MEMORY; break; } // // We use a direct mapping to find the immediately // the buffer headers. Unallocated pages are in a single entry // link list in that table. We must remove the locked entry // from the link list and save the buffer header address to the // new slot. The offset of the entry defines also the free // unlocked buffer in the buffer pool. // I have used this funny structure to minimize header // information for the unlocked virtual pages (you could have // a huge virtual buffer pool with a very small overhead in DLC). // FreeSlotIndex = (UINT)(((ULONG_PTR)pBufferPool->pUnlockedEntryList - (ULONG_PTR)pBufferPool->BufferHeaders) / sizeof(PVOID)); pBuffer->Header.BufferState = BUF_READY; pBuffer->Header.pLocalVa = (PVOID)((PCHAR)pBufferPool->BaseOffset + FreeSlotIndex * MAX_DLC_BUFFER_SEGMENT); pBufferPool->pUnlockedEntryList = pBufferPool->pUnlockedEntryList->pNext; pBufferPool->BufferHeaders[FreeSlotIndex] = pBuffer; // // Lock memory always outside the spin locks on 0 level. // UNLOCK_BUFFER_POOL(); RELEASE_DRIVER_LOCK(); pBuffer->Header.pMdl = AllocateProbeAndLockMdl(pBuffer->Header.pLocalVa, MAX_DLC_BUFFER_SEGMENT ); ACQUIRE_DRIVER_LOCK(); LOCK_BUFFER_POOL(); if (pBuffer->Header.pMdl) { pBuffer->Header.pGlobalVa = MmGetSystemAddressForMdl(pBuffer->Header.pMdl); pBuffer->Header.FreeSegments = MAX_DLC_BUFFER_SEGMENT / MIN_DLC_BUFFER_SEGMENT; status = AllocateBufferHeader( #if DBG pFileContext, #endif pBufferPool, pBuffer, MAX_DLC_BUFFER_SEGMENT / MIN_DLC_BUFFER_SEGMENT, 0, // logical index within the page 0 // page in the free page table ); } else { MemoryLockFailed = TRUE; status = DLC_STATUS_MEMORY_LOCK_FAILED; #if DBG DbgPrint("DLC.BufferPoolExpand: AllocateProbeAndLockMdl(a=%x, l=%x) failed\n", pBuffer->Header.pLocalVa, MAX_DLC_BUFFER_SEGMENT ); #endif } if (status != STATUS_SUCCESS) { // // It failed => free MDL (if non-null) and // restore the link list of available buffers // if (pBuffer->Header.pMdl != NULL) { UnlockAndFreeMdl(pBuffer->Header.pMdl); } pBufferPool->BufferHeaders[FreeSlotIndex] = pBufferPool->pUnlockedEntryList; pBufferPool->pUnlockedEntryList = (PDLC_BUFFER_HEADER)&(pBufferPool->BufferHeaders[FreeSlotIndex]); DEALLOCATE_PACKET_DLC_BUF(pBufferPool->hHeaderPool, pBuffer); break; } #if LLC_DBG CHECK_FREE_SEGMENT_COUNT(pBuffer->Header.pNextChild); #endif pBufferPool->FreeSpace += MAX_DLC_BUFFER_SEGMENT; pBufferPool->UncommittedSpace += MAX_DLC_BUFFER_SEGMENT; pBufferPool->BufferPoolSize += MAX_DLC_BUFFER_SEGMENT; pBufferPool->MissingSize -= MAX_DLC_BUFFER_SEGMENT; LlcInsertTailList(&pBufferPool->PageHeaders, pBuffer); } pBufferPool->MissingSize = 0; // // We will return success, if at least the minimal amount // memory was allocated. The initial pool size may be too // big for the current memory constraints set by the // operating system and actual available physical memory. // if (pBufferPool->UncommittedSpace < 0) { status = DLC_STATUS_NO_MEMORY; } } UNLOCK_BUFFER_POOL(); return status; } VOID BufferPoolFreeExtraPages( #if DBG IN PDLC_FILE_CONTEXT pFileContext, #endif IN PDLC_BUFFER_POOL pBufferPool ) /*++ Routine Description: The function checks the maximum Thresholds and unlocks the extra pages and deallocates their buffer headers. Arguments: pBufferPool - handle of buffer pool data structure. Return Value: None. --*/ { PDLC_BUFFER_HEADER pBuffer; KIRQL irql; PDLC_BUFFER_HEADER pNextBuffer; ASSUME_IRQL(DISPATCH_LEVEL); /* DbgPrint("MaxBufferSize: %x\n", pBufferPool->MaxBufferSize); DbgPrint("Uncommitted size: %x\n", pBufferPool->UncommittedSpace); DbgPrint("BufferPoolSize: %x\n", pBufferPool->BufferPoolSize); DbgPrint("FreeSpace : %x\n", pBufferPool->FreeSpace); */ LOCK_BUFFER_POOL(); // // Free the extra pages until we have enough free buffer space. // pBuffer = (PDLC_BUFFER_HEADER)pBufferPool->PageHeaders.Flink; while ((pBufferPool->UncommittedSpace > MAX_FREE_SIZE_THRESHOLD) && (pBuffer != (PVOID)&pBufferPool->PageHeaders)) { // // We may free (unlock) only those buffers given, that have // all buffers free. // if ((UINT)(pBuffer->Header.FreeSegments == (MAX_DLC_BUFFER_SEGMENT / MIN_DLC_BUFFER_SEGMENT))) { pNextBuffer = pBuffer->Header.pNextHeader; #if DBG DeallocateBuffer(pFileContext, pBufferPool, pBuffer); #else DeallocateBuffer(pBufferPool, pBuffer); #endif pBufferPool->FreeSpace -= MAX_DLC_BUFFER_SEGMENT; pBufferPool->UncommittedSpace -= MAX_DLC_BUFFER_SEGMENT; pBufferPool->BufferPoolSize -= MAX_DLC_BUFFER_SEGMENT; pBuffer = pNextBuffer; } else { pBuffer = pBuffer->Header.pNextHeader; } } UNLOCK_BUFFER_POOL(); } VOID DeallocateBuffer( #if DBG IN PDLC_FILE_CONTEXT pFileContext, #endif IN PDLC_BUFFER_POOL pBufferPool, IN PDLC_BUFFER_HEADER pBuffer ) /*++ Routine Description: The routine unlinks all segments of a page from the free lists and deallocates the data structures. Arguments: pBufferPool - handle of buffer pool data structure. pBuffer - the deallocated buffer header Return Value: None --*/ { UINT FreeSlotIndex; PDLC_BUFFER_HEADER pSegment, pNextSegment; // // First we unlink the segments from the free lists and // then free and unlock the data structs of segment. // for (pSegment = pBuffer->Header.pNextChild; pSegment != NULL; pSegment = pNextSegment) { pNextSegment = pSegment->FreeBuffer.pNextChild; // // Remove the buffer from the free lists (if it is there) // if (pSegment->FreeBuffer.BufferState == BUF_READY) { LlcRemoveEntryList(pSegment); } #if LLC_DBG else { // // This else can be possible only if we are // deleting the whole buffer pool (ref count=0) // if (pBufferPool->ReferenceCount != 0) { DbgPrint("Error: Invalid buffer state!"); DbgBreakPoint(); } pSegment->FreeBuffer.pNext = NULL; } #endif IoFreeMdl(pSegment->FreeBuffer.pMdl); DBG_INTERLOCKED_DECREMENT(AllocatedMdlCount); DEALLOCATE_PACKET_DLC_BUF(pBufferPool->hHeaderPool, pSegment); } // // Link the page to the free page list in buffer pool header // FreeSlotIndex = (UINT)(((ULONG_PTR)pBuffer->Header.pLocalVa - (ULONG_PTR)pBufferPool->BaseOffset) / MAX_DLC_BUFFER_SEGMENT); pBufferPool->BufferHeaders[FreeSlotIndex] = pBufferPool->pUnlockedEntryList; pBufferPool->pUnlockedEntryList = (PDLC_BUFFER_HEADER)&(pBufferPool->BufferHeaders[FreeSlotIndex]); UnlockAndFreeMdl(pBuffer->Header.pMdl); LlcRemoveEntryList(pBuffer); DEALLOCATE_PACKET_DLC_BUF(pBufferPool->hHeaderPool, pBuffer); } NTSTATUS AllocateBufferHeader( #if DBG IN PDLC_FILE_CONTEXT pFileContext, #endif IN PDLC_BUFFER_POOL pBufferPool, IN PDLC_BUFFER_HEADER pParent, IN UCHAR Size, IN UCHAR Index, IN UINT FreeListTableIndex ) /*++ Routine Description: The routine allocates and initializes a new buffer segment and links it to the given free segment list. Arguments: pBufferPool - handle of buffer pool data structure. pParent - the parent (page) node of this segemnt Size - size of this segment in 256 byte units Index - index of this segment in 256 byte units FreeListTableIndex - log2(Size), (ie. 256 bytes=>0, etc.) Return Value: Returns NTSTATUS Success - STATUS_SUCCESS Failure - DLC_STATUS_NO_MEMORY --*/ { PDLC_BUFFER_HEADER pBuffer; ASSUME_IRQL(DISPATCH_LEVEL); if (!(pBuffer = ALLOCATE_PACKET_DLC_BUF(pBufferPool->hHeaderPool))) { return DLC_STATUS_NO_MEMORY; } pBuffer->FreeBuffer.pMdl = IoAllocateMdl((PUCHAR)pParent->Header.pLocalVa + (UINT)Index * MIN_DLC_BUFFER_SEGMENT, (UINT)Size * MIN_DLC_BUFFER_SEGMENT, FALSE, // not used (no IRP) FALSE, // we can't take this from user quota NULL ); if (pBuffer->FreeBuffer.pMdl == NULL) { DEALLOCATE_PACKET_DLC_BUF(pBufferPool->hHeaderPool, pBuffer); return DLC_STATUS_NO_MEMORY; } DBG_INTERLOCKED_INCREMENT(AllocatedMdlCount); pBuffer->FreeBuffer.pNextChild = pParent->Header.pNextChild; pParent->Header.pNextChild = pBuffer; pBuffer->FreeBuffer.pParent = pParent; pBuffer->FreeBuffer.Size = Size; pBuffer->FreeBuffer.Index = Index; pBuffer->FreeBuffer.BufferState = BUF_READY; pBuffer->FreeBuffer.FreeListIndex = (UCHAR)FreeListTableIndex; // // Link the full page buffer to the first free list // LlcInsertHeadList(&(pBufferPool->FreeLists[FreeListTableIndex]), pBuffer); return STATUS_SUCCESS; } NTSTATUS BufferPoolAllocate( #if DBG IN PDLC_FILE_CONTEXT pFileContext, #endif IN PDLC_BUFFER_POOL pBufferPool, IN UINT BufferSize, IN UINT FrameHeaderSize, IN UINT UserDataSize, IN UINT FrameLength, IN UINT SegmentSizeIndex, IN OUT PDLC_BUFFER_HEADER *ppBufferHeader, OUT PUINT puiBufferSizeLeft ) /*++ Routine Description: Function allocates the requested buffer (locked and mapped) from the buffer pool and returns its MDL and descriptor table of user segments. The returned buffer is actually the minimal combination of some segments (256, 512, 1024, 2048, 4096). There is a header in each buffer segment. The size of the frame header and the user data added to all frames are defined by the caller. The allocated segments will be linked in three level: - Segment headers will be linked in the reserved list to be checked when the application program releases the buffers back to pool. The actual segments cannot be used, because they are located in unsafe user memory. - Segments are linked (from smaller to bigger) for application program, this link list is used nowhere in the driver (because it is in ...) - MDLs of the same buffer list are linked for the driver The link lists goes from smaller segment to bigger, because the last segment should be the biggest one in a transmit call (it actually works very nice with 2 or 4 token ring frames). DON'T TOUCH the calculation of the segment size (includes operations with BufferSize, ->Cont.DataLength and FirstHeaderSize), the logic is very complex. The current code has been tested by hand using some test values, and it seems to work (BufferSize = 0, 1, 0xF3, FirstHeader = 0,2). Arguments: pBufferPool - handle of buffer pool data structure. BufferSize - the size of the actual data in the requested buffers. This must really be the actual data. Nobody can know the size of all segment headers beforehand. The buffer size must include the frame header size added to the first buffer in the list! FrameHeaderSize - the space reserved for the frame header depends on buffer format (OS/2 or DOS) and if the data is read contiguously or not. The buffer manager reserves four bytes from the beginning of the first segment in frame to link this frame to next frames. UserDataSize - buffer area reserved for user data (nobody uses this) FrameLength - the total frame length (may not be the same as BufferSize, because the LAN and DLC headers may be saved into the header. SegmentSizeIndex - the client may ask a number segments having a fixed size (256, 512, ... 4096). ppBufferHeader - parameter returns the arrays of the user buffer segments. The array is allocated in the end of this buffer. This may include a pointer to a buffer pool, that is already allocated. The old buffer list will be linked behind the new buffers. puiBufferSizeLeft - returns the size of buffer space, that is not yet allocated. The client may extend the buffer pool and then continue the allocation of the buffers. Otherwise you could not allocate more buffers than defined by MinFreeSizeThreshold. Return Value: NTSTATUS STATUS_SUCCESS DLC_STATUS_NO_MEMORY - no available memory in the non paged pool --*/ { INT i, j, k; // loop indexes (three level loop) INT LastIndex; // index of smallest allowed segment size. INT LastAvailable; // Index of free list having biggest segments UINT SegmentSize; // current segment size PDLC_BUFFER_HEADER pPrev; PMDL pPrevMdl; PDLC_BUFFER_HEADER pNew; PFIRST_DLC_SEGMENT pDlcBuffer, pLastDlcBuffer; NTSTATUS Status = STATUS_SUCCESS; KIRQL irql; USHORT SavedDataLength; static USHORT SegmentSizes[DLC_BUFFER_SEGMENTS] = { #if defined(ALPHA) 8192, #endif 4096, 2048, 1024, 512, 256 }; ASSUME_IRQL(DISPATCH_LEVEL); // // Link the old buffers behind the new ones. // This is really sick: BufferGet is calling this second (or more) // time after it has expanded the buffer pool for the new retry, // we must search the last buffer header, because the extra // buffer space is removed from it. // pPrev = *ppBufferHeader; if (pPrev != NULL) { for (pNew = pPrev; pNew->FrameBuffer.pNextSegment != NULL; pNew = pNew->FrameBuffer.pNextSegment) { ; // NOP } pLastDlcBuffer = (PFIRST_DLC_SEGMENT) ( (PUCHAR)pNew->FreeBuffer.pParent->Header.pGlobalVa + (UINT)pNew->FreeBuffer.Index * MIN_DLC_BUFFER_SEGMENT ); } // // the first frame size has been added to the total length // (excluding the default header), but we must // exclude the default buffer header. // if (FrameHeaderSize > sizeof(NEXT_DLC_SEGMENT)) { FrameHeaderSize -= sizeof(NEXT_DLC_SEGMENT); } else { FrameHeaderSize = 0; } // // The frame header must be included in the total buffer space // just as any other stuff. We must add the maximum extra size // to get all stuff to fit into buffers. // BufferSize += MIN_DLC_BUFFER_SEGMENT - 1 + FrameHeaderSize; // // Initialize the index variables for the loop // if (SegmentSizeIndex == -1) { i = 0; LastIndex = DLC_BUFFER_SEGMENTS - 1; SegmentSize = MAX_DLC_BUFFER_SEGMENT; } else { i = SegmentSizeIndex; LastIndex = SegmentSizeIndex; SegmentSize = SegmentSizes[SegmentSizeIndex]; } LastAvailable = 0; LOCK_BUFFER_POOL(); // // Loop until we have found enough buffers for // the given buffer space (any kind, but as few as possible) // or for the given number of requested buffers. // Initialize each new buffer. The frame header is a special case. // We go from bigger segments to smaller ones. The last (and smallest) // will be initialized as a frame header (if needed). // for (; (i <= LastIndex) && BufferSize; i++) { while (((SegmentSize - sizeof(NEXT_DLC_SEGMENT) - UserDataSize) < BufferSize) || (i == LastIndex)) { // // Check if there are any buffers having the optimal size // if (IsListEmpty(&pBufferPool->FreeLists[i])) { // // Split a bigger segment to smallers. Link the // extra segments to the free lists and return // after that to the current size level. // for (j = i; j > LastAvailable; ) { j--; if (!IsListEmpty(&pBufferPool->FreeLists[j])) { // // Take the first available segment header in // the free list // pNew = LlcRemoveHeadList(&pBufferPool->FreeLists[j]); // // Split segments until we reach the desired level. // We leave every (empty) level between a new segment // header (including the current level (= i). // k = j; do { k++; // // We must also split the orginal buffer header // and its MDL. // pNew->FreeBuffer.Size /= 2; pNew->FreeBuffer.FreeListIndex++; // // We create the new buffer header for // the upper half of the old buffer segment. // Status = AllocateBufferHeader( #if DBG pFileContext, #endif pBufferPool, pNew->FreeBuffer.pParent, pNew->FreeBuffer.Size, (UCHAR)(pNew->FreeBuffer.Index + pNew->FreeBuffer.Size), (UINT)k ); // // We cannot stop on error, but we try to // allocate several smaller segments before // we will give up. // if (Status != STATUS_SUCCESS) { // // We couldn't split the buffer, return // the current buffer back to its slot. // pNew->FreeBuffer.Size *= 2; pNew->FreeBuffer.FreeListIndex--; LlcInsertHeadList(&pBufferPool->FreeLists[k-1], pNew ); break; } } while (k != i); break; } } // // Did we succeed to split the bigger segments // to smaller ones? // if (IsListEmpty(&pBufferPool->FreeLists[i])) { // // We have run out of bigger segments, let's try to // use the smaller ones instead. Indicate, that // there exist no bigger segments than current one. // THIS BREAK STARTS A NEW LOOP WITH A SMALLER // SEGMENT SIZE. // LastAvailable = i; break; } } else { pNew = LlcRemoveHeadList(&pBufferPool->FreeLists[i]); } pDlcBuffer = (PFIRST_DLC_SEGMENT) ((PUCHAR)pNew->FreeBuffer.pParent->Header.pGlobalVa + (UINT)pNew->FreeBuffer.Index * MIN_DLC_BUFFER_SEGMENT); // // The buffers must be chained together on three level: // - using kernel Buffer headers (for driver) // - by user pointer (for apps) // - MDLs (for NDIS) // if (pPrev == NULL) { // // Frame header - initialize the list // HACK-HACK!!!! // pPrevMdl = NULL; pDlcBuffer->Cont.pNext = NULL; pLastDlcBuffer = pDlcBuffer; } else { pPrevMdl = pPrev->FrameBuffer.pMdl; pDlcBuffer->Cont.pNext = (PNEXT_DLC_SEGMENT) ((PUCHAR)pPrev->FrameBuffer.pParent->Header.pLocalVa + (UINT)pPrev->FrameBuffer.Index * MIN_DLC_BUFFER_SEGMENT); } pBufferPool->FreeSpace -= SegmentSize; pBufferPool->UncommittedSpace -= SegmentSize; pNew->FrameBuffer.pNextFrame = NULL; pNew->FrameBuffer.BufferState = BUF_USER; pNew->FrameBuffer.pNextSegment = pPrev; pNew->FrameBuffer.pParent->Header.FreeSegments -= pNew->FreeBuffer.Size; #if LLC_DBG if ((UINT)(MIN_DLC_BUFFER_SEGMENT * pNew->FreeBuffer.Size) != SegmentSize) { DbgPrint("Invalid buffer size.\n"); DbgBreakPoint(); } CHECK_FREE_SEGMENT_COUNT(pNew); #endif pPrev = pNew; pDlcBuffer->Cont.UserOffset = sizeof(NEXT_DLC_SEGMENT); pDlcBuffer->Cont.UserLength = (USHORT)UserDataSize; pDlcBuffer->Cont.FrameLength = (USHORT)FrameLength; // Save this length in a local var since pDlcBuffer->Cont.DataLength can be changed by user // but this is used later on also. SavedDataLength = (USHORT)(SegmentSize - sizeof(NEXT_DLC_SEGMENT) - UserDataSize); pDlcBuffer->Cont.DataLength = SavedDataLength; // // Check if we have done it! // Remember, that the buffer size have been round up/over to // the next 256 bytes even adderss => we never go negative. // // 127041: User can change this value between this and the last instruction //BufferSize -= pDlcBuffer->Cont.DataLength; BufferSize -= SavedDataLength; if (BufferSize < MIN_DLC_BUFFER_SEGMENT) { pDlcBuffer->Cont.UserOffset += (USHORT)FrameHeaderSize; pDlcBuffer->Cont.DataLength -= (USHORT)FrameHeaderSize; SavedDataLength -= (USHORT)FrameHeaderSize; // // The data must be read to the beginning of the // buffer chain (eg. because of NdisTransferData). // => the first buffer must be full and the last // one must always be odd. The extra length // in the partial MDL does not matter. // BufferSize -= MIN_DLC_BUFFER_SEGMENT - 1; pLastDlcBuffer->Cont.DataLength += (USHORT)BufferSize; BuildMappedPartialMdl( pNew->FrameBuffer.pParent->Header.pMdl, pNew->FrameBuffer.pMdl, pNew->FrameBuffer.pParent->Header.pLocalVa + pNew->FrameBuffer.Index * MIN_DLC_BUFFER_SEGMENT + FrameHeaderSize + UserDataSize + sizeof(NEXT_DLC_SEGMENT), SavedDataLength ); pNew->FrameBuffer.pMdl->Next = pPrevMdl; // // The buffer headers must be procted (the flag prevents // user to free them back buffer pool before we have // indicated the chained receive frames to him). // The linkage of frame headers will crash, if // the header buffer is released before the frame // was indicated! // pNew->FrameBuffer.BufferState = BUF_RCV_PENDING; BufferSize = 0; break; } else { // // MDL must exclude the buffer header from the actual data. // BuildMappedPartialMdl( pNew->FrameBuffer.pParent->Header.pMdl, pNew->FrameBuffer.pMdl, pNew->FrameBuffer.pParent->Header.pLocalVa + pNew->FrameBuffer.Index * MIN_DLC_BUFFER_SEGMENT + UserDataSize + sizeof(NEXT_DLC_SEGMENT), pDlcBuffer->Cont.DataLength ); pNew->FrameBuffer.pMdl->Next = pPrevMdl; } } SegmentSize /= 2; } if (BufferSize == 0) { Status = STATUS_SUCCESS; } else { BufferSize -= (MIN_DLC_BUFFER_SEGMENT - 1); // // The client, that is not running in DPC level may extend // the buffer pool, if there is still available space left // in the buffer pool // if (pBufferPool->MaxBufferSize > pBufferPool->BufferPoolSize) { // // We can expand the buffer pool, sometimes we must // allocate new bigger segments, if the available // smaller segments cannot satisfy the request. // if ((LONG)BufferSize > pBufferPool->MissingSize) { pBufferPool->MissingSize = (LONG)BufferSize; } Status = DLC_STATUS_EXPAND_BUFFER_POOL; } else { Status = DLC_STATUS_INADEQUATE_BUFFERS; } } UNLOCK_BUFFER_POOL(); *ppBufferHeader = pPrev; *puiBufferSizeLeft = BufferSize; return Status; } NTSTATUS BufferPoolDeallocate( IN PDLC_BUFFER_POOL pBufferPool, IN UINT BufferCount, IN PLLC_TRANSMIT_DESCRIPTOR pBuffers ) /*++ Routine Description: Function deallocates the requested buffers. It first checks the user buffer in the page table and then adds its header to the free list. Arguments: pBufferPool - handle of buffer pool data structure. BufferCount - number of user buffers to be released pBuffers - array of the user buffers Return Value: NTSTATUS --*/ { PDLC_BUFFER_HEADER pBuffer; UINT i; NTSTATUS status = STATUS_SUCCESS; KIRQL irql; ASSUME_IRQL(PASSIVE_LEVEL); LOCK_BUFFER_POOL(); // // Return all buffers // for (i = 0; i < BufferCount; i++) { pBuffer = GetBufferHeader(pBufferPool, pBuffers[i].pBuffer); if (pBuffer && (pBuffer->FreeBuffer.BufferState == BUF_USER)) { register ULONG bufsize; // // Set the buffer state READY and restore the modified // size and offset fields in MDL // pBuffer->FreeBuffer.BufferState = BUF_READY; pBuffer->FreeBuffer.pParent->Header.FreeSegments += pBuffer->FreeBuffer.Size; #if LLC_DBG if (pBuffer->FreeBuffer.pParent->Header.FreeSegments > 16) { DbgPrint("Invalid buffer size.\n"); DbgBreakPoint(); } CHECK_FREE_SEGMENT_COUNT(pBuffer); #endif // // a microscopic performance improvement: the compiler (x86 at // least) generates the sequence of instructions to work out // the buffer size (number of blocks * block size) TWICE, // presumably because it can't assume that the structure hasn't // changed between the 2 accesses? Anyhow, Nature abhors a // vacuum, which is why my house is such a mess // bufsize = pBuffer->FreeBuffer.Size * MIN_DLC_BUFFER_SEGMENT; pBufferPool->FreeSpace += bufsize; pBufferPool->UncommittedSpace += bufsize; LlcInsertTailList(&pBufferPool->FreeLists[pBuffer->FreeBuffer.FreeListIndex], pBuffer); } else { // // At least one of the released buffers is invalid, // may be already released, or it may not exist in // the buffer pool at all // status = DLC_STATUS_INVALID_BUFFER_ADDRESS; } } UNLOCK_BUFFER_POOL(); return status; } VOID BufferPoolDeallocateList( IN PDLC_BUFFER_POOL pBufferPool, IN PDLC_BUFFER_HEADER pBufferList ) /*++ Routine Description: Function deallocates the requested buffer list. The buffer list may be circular or null terminated. Arguments: pBufferPool - handle of buffer pool data structure. pBufferList - link list of user Return Value: None --*/ { PDLC_BUFFER_HEADER pBuffer, pNextBuffer, pFrameBuffer, pNextFrameBuffer; KIRQL irql; if (pBufferList == NULL) { return; } LOCK_BUFFER_POOL(); // // Return all buffers to the free lists. // The segments are always linked to a null terminated link list. // The frames are linked either circular or null terminated // link list! // // Note: both next segment and frame pointers are overlayed with // the pPrev and pNext pointers of the double linked free lists. // pNextFrameBuffer = pBufferList; do { pBuffer = pFrameBuffer = pNextFrameBuffer; pNextFrameBuffer = pFrameBuffer->FrameBuffer.pNextFrame; do { pNextBuffer = pBuffer->FrameBuffer.pNextSegment; #if LLC_DBG if (pBuffer->FreeBuffer.BufferState != BUF_USER && pBuffer->FreeBuffer.BufferState != BUF_RCV_PENDING) { DbgBreakPoint(); } if (pBuffer->FreeBuffer.pParent->Header.FreeSegments > 16) { DbgPrint("Invalid buffer size.\n"); DbgBreakPoint(); } CHECK_FREE_SEGMENT_COUNT(pBuffer); #endif // // Set the buffer state READY and restore the modified // size and offset fields in MDL // pBuffer->FreeBuffer.BufferState = BUF_READY; pBuffer->FreeBuffer.pParent->Header.FreeSegments += pBuffer->FreeBuffer.Size; pBufferPool->FreeSpace += pBuffer->FreeBuffer.Size * MIN_DLC_BUFFER_SEGMENT; pBufferPool->UncommittedSpace += pBuffer->FreeBuffer.Size * MIN_DLC_BUFFER_SEGMENT; LlcInsertTailList(&pBufferPool->FreeLists[pBuffer->FreeBuffer.FreeListIndex], pBuffer ); } while ( pBuffer = pNextBuffer ); } while (pNextFrameBuffer && (pNextFrameBuffer != pBufferList)); UNLOCK_BUFFER_POOL(); } NTSTATUS BufferPoolBuildXmitBuffers( IN PDLC_BUFFER_POOL pBufferPool, IN UINT BufferCount, IN PLLC_TRANSMIT_DESCRIPTOR pBuffers, IN OUT PDLC_PACKET pPacket ) /*++ Routine Description: Function build a MDL and buffer header list for a frame defined by a scatter/gather array. All buffers outside the buffer pool are probed and locked. All MDLs (the locked and ones for buffer pool) are chained together. The buffer pool headers are also chained. If any errors have been found the buffers are released using the reverse function (BufferPoolFreeXmitBuffers). THIS FUNCTION HAS A VERY SPECIAL SPIN LOCKING DESIGN: First we free the global spin lock (and lower the IRQ level to the lowest), Then, if the transmit is made from DLC buffers, we lock the spin lock again using NdisSpinLock function, that saves and restores the IRQL level, when it acquires and releases the spin lock. This all is done to minimize the spin locking overhead when we are locking transmit buffers, that are usually DLC buffers or normal user memory but not both. Arguments: pBufferPool - handle of buffer pool data structure, THIS MAY BE NULL!!!! BufferCount - number of user buffers in the frame pBuffers - array of the user buffers of the frame pPacket - generic DLC packet used in transmit Return Value: NTSTATUS --*/ { PDLC_BUFFER_HEADER pBuffer, pPrevBuffer = NULL; PMDL pMdl, pPrevMdl = NULL; INT i; NTSTATUS Status = STATUS_SUCCESS; BOOLEAN FirstBuffer = TRUE; UINT InfoFieldLength = 0; BOOLEAN BufferPoolIsLocked = FALSE; // very neat optimization! KIRQL irql; ASSUME_IRQL(PASSIVE_LEVEL); // // The client may allocate buffer headers without any buffers! // if (BufferCount != 0) { // // walk the buffers in a reverse order to build the // list in a convinient way. // for (i = BufferCount - 1; i >= 0; i--) { if (pBuffers[i].cbBuffer == 0) { continue; } InfoFieldLength += pBuffers[i].cbBuffer; // // Check first if the given address is in the same area as the // buffer pool // if (pBufferPool != NULL && (ULONG_PTR)pBuffers[i].pBuffer >= (ULONG_PTR)pBufferPool->BaseOffset && (ULONG_PTR)pBuffers[i].pBuffer < (ULONG_PTR)pBufferPool->MaxOffset) { // // Usually all transmit buffers are either in the buffer // pool or they are elsewhere in the user memory. // This boolean flag prevents us to toggle the buffer // pool spinlock for each transmit buffer segment. // (and nt spinlock is slower than its critical section!!!) // if (BufferPoolIsLocked == FALSE) { LOCK_BUFFER_POOL(); BufferPoolIsLocked = TRUE; } } // // The previous check does not yet garantee, that the given buffer // if really a buffer pool segment, but the buffer pool is // is now unlocked if it was aboslutely outside of the buffer pool, // GetBufferHeader- function requires, that the buffer pool // is locked, when it is called! // if (BufferPoolIsLocked && (pBuffer = GetBufferHeader(pBufferPool, pBuffers[i].pBuffer)) != NULL) { // // The provided buffer must be inside the allocated // buffer, otherwise the user has corrupted its buffers. // user offset within buffer + user length <= buffer // length // The buffer must be also be owned by the user // if (((ULONG_PTR)pBuffers[i].pBuffer & (MIN_DLC_BUFFER_SEGMENT - 1)) + (ULONG)pBuffers[i].cbBuffer > (ULONG)(pBuffer->FreeBuffer.Size * MIN_DLC_BUFFER_SEGMENT) || (pBuffer->FrameBuffer.BufferState & BUF_USER) == 0) { Status = DLC_STATUS_BUFFER_SIZE_EXCEEDED; break; } // // The same DLC buffer may be referenced several times. // Create a partial MDL for it and add the reference // counter. // if (pBuffer->FrameBuffer.BufferState & BUF_LOCKED) { pMdl = IoAllocateMdl(pBuffers[i].pBuffer, pBuffers[i].cbBuffer, FALSE, // not used (no IRP) FALSE, // can't charge from quota now NULL // Do not link it to IRPs ); if (pMdl == NULL) { Status = DLC_STATUS_NO_MEMORY; break; } DBG_INTERLOCKED_INCREMENT(AllocatedMdlCount); BuildMappedPartialMdl(pBuffer->FrameBuffer.pParent->Header.pMdl, pMdl, pBuffers[i].pBuffer, pBuffers[i].cbBuffer ); pBuffer->FrameBuffer.ReferenceCount++; if (pBuffers[i].boolFreeBuffer) { pBuffer->FrameBuffer.BufferState |= DEALLOCATE_AFTER_USE; } } else { // // Modify the MDL for this request, the length must // not be bigger than the buffer length and the // offset must be within the first 255 bytes of // the buffer. Build also the buffer header list // (i don't know why?) // pMdl = pBuffer->FrameBuffer.pMdl; if ( ((UINT)(ULONG_PTR)pBuffers[i].pBuffer & (MIN_DLC_BUFFER_SEGMENT - 1)) + pBuffers[i].cbBuffer > (UINT)pBuffer->FrameBuffer.Size * MIN_DLC_BUFFER_SEGMENT) { Status = DLC_STATUS_INVALID_BUFFER_LENGTH; break; } pBuffer->FrameBuffer.pNextSegment = pPrevBuffer; pBuffer->FrameBuffer.BufferState |= BUF_LOCKED; pBuffer->FrameBuffer.ReferenceCount = 1; if (pBuffers[i].boolFreeBuffer) { pBuffer->FrameBuffer.BufferState |= DEALLOCATE_AFTER_USE; } pPrevBuffer = pBuffer; // // DLC applications may change the user length or // buffer length of the frames given to them => // we must reinitialize global buffer and its length // BuildMappedPartialMdl(pBuffer->FrameBuffer.pParent->Header.pMdl, pMdl, pBuffers[i].pBuffer, pBuffers[i].cbBuffer ); } } else { if (BufferPoolIsLocked == TRUE) { UNLOCK_BUFFER_POOL(); BufferPoolIsLocked = FALSE; } // // Setup the exception handler around the memory manager // calls and clean up any extra data if this fails. // pMdl = AllocateProbeAndLockMdl(pBuffers[i].pBuffer, pBuffers[i].cbBuffer); if (pMdl == NULL) { Status = DLC_STATUS_MEMORY_LOCK_FAILED; #if DBG DbgPrint("DLC.BufferPoolBuildXmitBuffers: AllocateProbeAndLockMdl(a=%x, l=%x) failed\n", pBuffers[i].pBuffer, pBuffers[i].cbBuffer ); #endif break; } #if LLC_DBG cLockedXmitBuffers++; #endif } // // Chain all MDLs together // pMdl->Next = pPrevMdl; pPrevMdl = pMdl; } } if (BufferPoolIsLocked == TRUE) { UNLOCK_BUFFER_POOL(); } pPacket->Node.pNextSegment = pPrevBuffer; pPacket->Node.pMdl = pPrevMdl; pPacket->Node.LlcPacket.InformationLength = (USHORT)InfoFieldLength; if (Status != STATUS_SUCCESS) { // // Free all allocated buffer (but the last one because there // was an error with it) // BufferPoolFreeXmitBuffers(pBufferPool, pPacket); } return Status; } VOID BufferPoolFreeXmitBuffers( IN PDLC_BUFFER_POOL pBufferPool, IN PDLC_PACKET pXmitNode ) /*++ Routine Description: Function unlocks the xmit buffers that are not in the buffer pool. The caller must use DeallocateBufferPool routine to and deallocates and the buffers are returned back to the pool. The function has to separate the MDLs of user buffers and buffer pool MDLs. Arguments: pBufferPool - handle of buffer pool data structure. pXmitNode - pointer to a structure, that includes the buffer header list, MDL chain or it chains serveral transmits nodes and IRP together. Return Value: None --*/ { PDLC_BUFFER_HEADER pBuffer; PDLC_BUFFER_HEADER pOtherBuffer = NULL; PDLC_BUFFER_HEADER pNextBuffer = NULL; PMDL pMdl, pNextMdl; KIRQL irql; #if LLC_DBG BOOLEAN FrameCounted = FALSE; #endif // // Free all DLC buffers and MDLs linked in the transmit node. // MDL list may be larger than the buffer header list. // if (pXmitNode != NULL) { if (pBufferPool != NULL) { LOCK_BUFFER_POOL(); } pBuffer = pXmitNode->Node.pNextSegment; for (pMdl = pXmitNode->Node.pMdl; pMdl != NULL; pMdl = pNextMdl) { pNextMdl = pMdl->Next; pMdl->Next = NULL; // // Unlock only those MDLs, that are outside the buffer pool. // if ((pBuffer == NULL || pBuffer->FrameBuffer.pMdl != pMdl) && (pOtherBuffer = GetBufferHeader(pBufferPool, MmGetMdlVirtualAddress(pMdl))) == NULL) { #if LLC_DBG cUnlockedXmitBuffers++; #endif UnlockAndFreeMdl(pMdl); } else { // // This pointer can be NULL only if the first condition // if the previous 'if statement' was true => this cannot // be an orginal buffer header. // if (pOtherBuffer != NULL) { // // This is not the first reference of the buffer pool // segment, but a partial MDL created by a new // reference to a buffer segment already in use. // Free the paritial MDL and setup the buffer // pointer for the next loop. // pNextBuffer = pBuffer; pBuffer = pOtherBuffer; pOtherBuffer = NULL; IoFreeMdl(pMdl); DBG_INTERLOCKED_DECREMENT(AllocatedMdlCount); } else if (pBuffer != NULL) { // // This is the orginal refence of the buffer pool // segment, we may advance also in the buffer header // link list. // pNextBuffer = pBuffer->FrameBuffer.pNextSegment; } // // The same DLC buffer may be referenced several times. // Decrement the reference counter and free the // list if this was the last released reference. // pBuffer->FrameBuffer.ReferenceCount--; if (pBuffer->FrameBuffer.ReferenceCount == 0) { if (pBuffer->FrameBuffer.BufferState & DEALLOCATE_AFTER_USE) { // // Set the buffer state READY and restore the modified // size and offset fields in MDL // pBuffer->FreeBuffer.BufferState = BUF_READY; pBuffer->FreeBuffer.pParent->Header.FreeSegments += pBuffer->FreeBuffer.Size; #if LLC_DBG if (pBuffer->FreeBuffer.pParent->Header.FreeSegments > 16) { DbgPrint("Invalid buffer size.\n"); DbgBreakPoint(); } CHECK_FREE_SEGMENT_COUNT(pBuffer); #endif pBufferPool->FreeSpace += (UINT)pBuffer->FreeBuffer.Size * MIN_DLC_BUFFER_SEGMENT; pBufferPool->UncommittedSpace += (UINT)pBuffer->FreeBuffer.Size * MIN_DLC_BUFFER_SEGMENT; LlcInsertTailList(&pBufferPool->FreeLists[pBuffer->FreeBuffer.FreeListIndex], pBuffer); #if LLC_DBG if (FrameCounted == FALSE) { FrameCounted = TRUE; cFramesReleased++; } #endif } else { pBuffer->FreeBuffer.BufferState = BUF_USER; } } pBuffer = pNextBuffer; } } if (pBufferPool != NULL) { UNLOCK_BUFFER_POOL(); } } } PDLC_BUFFER_HEADER GetBufferHeader( IN PDLC_BUFFER_POOL pBufferPool, IN PVOID pUserBuffer ) /*++ Routine Description: Function returns the buffer pool header of the given buffer in the user address space or NULL, if the given address has no buffer. Arguments: pBufferPool - handle of buffer pool data structure. pUserBuffer - DLC buffer address in user memory Return Value: Pointer of DLC buffer header or NULL (if not found) --*/ { UINT PageTableIndex; UINT IndexWithinPage; PDLC_BUFFER_HEADER pBuffer; // // The buffer pool may not exist, when we are transmitting frames. // if (pBufferPool == NULL) { return NULL; } PageTableIndex = (UINT)(((ULONG_PTR)pUserBuffer - (ULONG_PTR)pBufferPool->BaseOffset) / MAX_DLC_BUFFER_SEGMENT); // // We simply discard the buffers outside the preallocated // virtual buffer in user space. We must also check, // that the buffer is really reserved and locked (ie. // it is not in the free list of unlocked entries). // Note, that the buffer pool base address have been aligned with // the maximum buffer segment size. // if (PageTableIndex >= (UINT)pBufferPool->MaximumIndex || ((ULONG_PTR)pBufferPool->BufferHeaders[PageTableIndex] >= (ULONG_PTR)pBufferPool->BufferHeaders && (ULONG_PTR)pBufferPool->BufferHeaders[PageTableIndex] < (ULONG_PTR)&pBufferPool->BufferHeaders[pBufferPool->MaximumIndex])) { return NULL; } IndexWithinPage = (UINT)(((ULONG_PTR)pUserBuffer & (MAX_DLC_BUFFER_SEGMENT - 1)) / MIN_DLC_BUFFER_SEGMENT); for ( pBuffer = pBufferPool->BufferHeaders[PageTableIndex]->Header.pNextChild; pBuffer != NULL; pBuffer = pBuffer->FreeBuffer.pNextChild) { if (pBuffer->FreeBuffer.Index == (UCHAR)IndexWithinPage) { // // We MUST not return a locked buffer, otherwise the app // will corrupt the whole buffer pool. // if ((pBuffer->FreeBuffer.BufferState & BUF_USER) == 0) { return NULL; } else { return pBuffer; } } } return NULL; } VOID BufferPoolDereference( #if DBG IN PDLC_FILE_CONTEXT pFileContext, #endif IN PDLC_BUFFER_POOL *ppBufferPool ) /*++ Routine Description: This routine decrements the reference count of the buffer pool and deletes it when the reference count hits to zero. Arguments: pFileContext - pointer to DLC_FILE_CONTEXT pBufferPool - opaque handle of buffer pool data structure. Return Value: None --*/ { PDLC_BUFFER_HEADER pBufferHeader, pNextHeader; KIRQL irql; PDLC_BUFFER_POOL pBufferPool = *ppBufferPool; ASSUME_IRQL(ANY_IRQL); *ppBufferPool = NULL; if (pBufferPool == NULL) { return; } LOCK_BUFFER_POOL(); if (pBufferPool->ReferenceCount != 0) { pBufferPool->ReferenceCount--; } if (pBufferPool->ReferenceCount == 0) { KIRQL Irql2; ACQUIRE_DLC_LOCK(Irql2); RemoveFromLinkList((PVOID*)&pBufferPools, pBufferPool); RELEASE_DLC_LOCK(Irql2); // // The buffer pool does not exist any more !!! // => we can remove the spin lock and free all resources // UNLOCK_BUFFER_POOL(); for (pBufferHeader = (PDLC_BUFFER_HEADER)pBufferPool->PageHeaders.Flink; !IsListEmpty(&pBufferPool->PageHeaders); pBufferHeader = pNextHeader) { pNextHeader = pBufferHeader->Header.pNextHeader; #if DBG DeallocateBuffer(pFileContext, pBufferPool, pBufferHeader); #else DeallocateBuffer(pBufferPool, pBufferHeader); #endif } DELETE_BUFFER_POOL_FILE(&pBufferPool->hHeaderPool); FREE_MEMORY_DRIVER(pBufferPool); } else { #if DBG DbgPrint("Buffer pool not released, reference count = %d\n", pBufferPool->ReferenceCount ); #endif UNLOCK_BUFFER_POOL(); } } NTSTATUS BufferPoolReference( IN HANDLE hExternalHandle, OUT PVOID *phOpaqueHandle ) /*++ Routine Description: This routine translates the the external buffer pool handle to a local opaque handle (=void pointer of the structure) and optioanlly checks the access rights of the current process to the buffer pool memory. The probing may raise an exeption to the IO- system, that will return error when this terminates. The function also increments the reference count of the buffer pool. Arguments: hExternalHandle - buffer handle allocated from the handle table phOpaqueHandle - opaque handle of buffer pool data structure Return Value: None --*/ { PDLC_BUFFER_POOL pBufferPool; NTSTATUS Status; KIRQL irql; ASSUME_IRQL(DISPATCH_LEVEL); ACQUIRE_DLC_LOCK(irql); for (pBufferPool = pBufferPools; pBufferPool != NULL; pBufferPool = pBufferPool->pNext) { if (pBufferPool == hExternalHandle) { break; } } RELEASE_DLC_LOCK(irql); if (pBufferPool == NULL) { return DLC_STATUS_INVALID_BUFFER_HANDLE; } // // We must do the optional probing outside of the spinlocks // and before we have incremented the reference count. // We do only read probing, because it is simpler. // RELEASE_DRIVER_LOCK(); Status = ProbeVirtualBuffer(pBufferPool->BaseOffset, pBufferPool->BufferPoolSize); ACQUIRE_DRIVER_LOCK(); if (Status == STATUS_SUCCESS) { LOCK_BUFFER_POOL(); pBufferPool->ReferenceCount++; *phOpaqueHandle = (PVOID)pBufferPool; UNLOCK_BUFFER_POOL(); } return Status; } NTSTATUS ProbeVirtualBuffer( IN PUCHAR pBuffer, IN LONG Length ) /*++ Routine Description: Tests an address range for accessability. Actually reads the first and last DWORDs in the address range, and assumes the rest of the memory is paged-in. Arguments: pBuffer - address to test Length - in bytes of region to check Return Value: NTSTATUS Success - STATUS_SUCCESS Failure - DLC_STATUS_MEMORY_LOCK_FAILED --*/ { NTSTATUS status = STATUS_SUCCESS; ASSUME_IRQL(PASSIVE_LEVEL); try { ProbeForRead(pBuffer, Length, sizeof(UCHAR)); } except(EXCEPTION_EXECUTE_HANDLER) { #if DBG DbgPrint("DLC.ProbeVirtualBuffer: Error: Can't ProbeForRead a=%x, l=%x\n", pBuffer, Length ); #endif status = DLC_STATUS_MEMORY_LOCK_FAILED; } return status; } PMDL AllocateProbeAndLockMdl( IN PVOID UserBuffer, IN UINT UserBufferLength ) /*++ Routine Description: This function just allocates, probes, locks and optionally maps any user buffer to kernel space. Returns NULL, if the operation fails for any reason. Remarks: This routine can be called only below DPC level and when the user context is known (ie. a spin locks must not be set!). Arguments: UserBuffer - user space address UserBufferLength - length of that buffer is user space Return Value: PMDL - pointer if successful NULL if not successful --*/ { PMDL pMdl; ASSUME_IRQL(PASSIVE_LEVEL); try { pMdl = IoAllocateMdl(UserBuffer, UserBufferLength, FALSE, // not used (no IRP) FALSE, // we don't charge the non-paged pool quota NULL // Do not link it to IRP ); if (pMdl != NULL) { #if DBG IF_DIAG(MDL_ALLOC) { PVOID caller, callerscaller; RtlGetCallersAddress(&caller, &callerscaller); DbgPrint("A: pMdl=%#x caller=%#x caller's=%#x\n", pMdl, caller, callerscaller ); } #endif DBG_INTERLOCKED_INCREMENT(AllocatedMdlCount); MmProbeAndLockPages(pMdl, UserMode, // Current user must have access! IoModifyAccess ); DBG_INTERLOCKED_ADD( LockedPageCount, +(ADDRESS_AND_SIZE_TO_SPAN_PAGES( ((ULONG)pMdl->StartVa | pMdl->ByteOffset), pMdl->ByteCount)) ); } } except(EXCEPTION_EXECUTE_HANDLER) { DBG_INTERLOCKED_INCREMENT(FailedMemoryLockings); if (pMdl != NULL) { IoFreeMdl(pMdl); DBG_INTERLOCKED_DECREMENT(AllocatedMdlCount); pMdl = NULL; } } return pMdl; } VOID BuildMappedPartialMdl( IN PMDL pSourceMdl, IN OUT PMDL pTargetMdl, IN PVOID BaseVa, IN ULONG Length ) /*++ Routine Description: This function builds a partial MDL from a mapped source MDL. The target MDL must have been initialized for the given size. The target MDL cannot be used after the source MDL has been unmapped. Remarks: MDL_PARTIAL_HAS_BEEN_MAPPED flag is not set in MdlFlag to prevent IoFreeMdl to unmap the virtual address. Arguments: pSourceMdl - Mapped source MDL pTargetMdl - Allocate MDL BaseVa - virtual base address Length - length of the data Return Value: None --*/ { ASSUME_IRQL(ANY_IRQL); if (Length) { LlcMemCpy(&pTargetMdl[1], &pSourceMdl[1], (UINT)(sizeof(ULONG) * ADDRESS_AND_SIZE_TO_SPAN_PAGES(BaseVa, Length)) ); } pTargetMdl->Next = NULL; pTargetMdl->StartVa = (PVOID)PAGE_ALIGN(BaseVa); pTargetMdl->ByteOffset = BYTE_OFFSET(BaseVa); pTargetMdl->ByteCount = Length; // // HACK-HACK-HACK-HACK-HACK-HACK-HACK-HACK-HACK-HACK-HACK-HACK // // The excellent NT memory manager doesn't provide any fast way to // create temporary MDLs that will be deallocated before their // actual source MDLs. // We will never map this MDL, because its mapped orginal source mdl // will be kept in memory until this (and its peers) have been // deallocated. // pTargetMdl->MdlFlags = (UCHAR)((pTargetMdl->MdlFlags & ~MDL_MAPPED_TO_SYSTEM_VA) | MDL_SOURCE_IS_NONPAGED_POOL); pTargetMdl->MappedSystemVa = (PVOID)((PCHAR)MmGetSystemAddressForMdl(pSourceMdl) + ((ULONG_PTR)BaseVa - (ULONG_PTR)MmGetMdlVirtualAddress(pSourceMdl))); } VOID UnlockAndFreeMdl( PMDL pMdl ) /*++ Routine Description: This function unmaps (if not a partial buffer), unlocks and and free a MDL. OK to call at DISPATCH_LEVEL Arguments: pMdl - pointer to MDL to free Return Value: None --*/ { ASSUME_IRQL(ANY_IRQL); DBG_INTERLOCKED_DECREMENT(AllocatedMdlCount); DBG_INTERLOCKED_ADD(LockedPageCount, -(ADDRESS_AND_SIZE_TO_SPAN_PAGES( ((ULONG)((PMDL)pMdl)->StartVa | ((PMDL)pMdl)->ByteOffset), (((PMDL)pMdl)->ByteCount))) ); MmUnlockPages((PMDL)pMdl); IoFreeMdl((PMDL)pMdl); #if DBG IF_DIAG(MDL_ALLOC) { PVOID caller, callerscaller; RtlGetCallersAddress(&caller, &callerscaller); DbgPrint("F: pMdl=%#x caller=%#x caller's=%#x\n", pMdl, caller, callerscaller ); } #endif }