From kanoj  Wed Feb  2 19:13:50 2000
Received: (from kanoj@localhost) by google.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) id TAA48607; Wed, 2 Feb 2000 19:13:49 -0800 (PST)
From: kanoj (Kanoj Sarcar)
Message-Id: <200002030313.TAA48607@google.engr.sgi.com>
Subject: Re: Memory problem
To: frankeh@us.ibm.com
Date: Wed, 2 Feb 2000 19:13:49 -0800 (PST)
Cc: kanoj (Kanoj Sarcar), rockhold@us.ibm.com
In-Reply-To: <85256874.007D310A.00@D51MTA07.pok.ibm.com> from "frankeh@us.ibm.com" at Jan 28, 2000 05:42:39 PM
X-Mailer: ELM [version 2.5 PL2]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Status: OR

Okay, I am attaching a patch here: I *think* this will allow all the
things that Hubertus wanted, without creating a new api 
__alloc_pages_zone(), which I doubt Linus will accept for 2.3.
The patch creates 2 new flag bits, __GFP_NOFALLBACK and 
__GFP_DISCONTIG. It is expected that mm/numa.c alloc_pages()
will set either/both of these bits when invoking alloc_pages_node()
according to what is needed. There is 0 impact on non-numa
platforms.

When alloc_pages() does not want to balance zones, it must clear
GFP_WAIT, and set __GFP_DISCONTIG. If the zone is unbalanced, 
kswapd will be poked, and  NULL returned from alloc_pages_node().
No fall back to lower zones will occur (other than due to race
conditions, ignore for the moment: or if you want to be ultra safe,
also pass in __GFP_NOFALLBACK).

When alloc_pages() is willing to balance zones (possibly incurring
io and waiting), it should not set __GFP_DISCONTIG. If it is not 
willing to satisy the request by falling back to a lower zone on the 
same node, pass in __GFP_NOFALLBACK. 

Note: handling of non GFP_WAIT requests (irrespective of 
GFP_HIGH/MED/LOW) should be done carefully. In the non-numa case, 
non GFP_WAIT requests try to blindly allocate from a zone, whether 
it is balanced or not, after poking kswapd. If such behavior is desired 
in the numa case, do not pass in __GFP_DISCONTIG. Your choice whether 
you want to pass in __GFP_NOFALLBACK. Make every attempt to allocate 
a page to a non GFP_WAIT request (if a free page exists), whether zones 
are balanced or not, after doing all your searches.

Hubertus, you also mentioned exposing try_to_free_pages(), that is 
already done. About exposing ZONE_BALANCED() and zone_balance_memory,
I did that when I sent a zone-aware kswapd patch to Linus, but he 
hasn't accepted it.

With this patch to the core routines, is there still some things that
a numa aware allocator can not achieve?

Kanoj

--- include/linux/mm.h	Tue Feb  1 14:24:59 2000
+++ include/linux/mm.h	Wed Feb  2 18:16:30 2000
@@ -461,6 +461,16 @@
 #endif
 
 #define __GFP_DMA	0x20
+#ifdef (CONFIG_HIGHMEM || CONFIG_DISCONTIGMEM)
+#define __GFP_NOFALLBACK 0x40	/* do not fall back on lower zones */
+#else /* CONFIG_HIGHMEM || CONFIG_DISCONTIGMEM */
+#define __GFP_NOFALLBACK 0x0
+#endif /* CONFIG_HIGHMEM || CONFIG_DISCONTIGMEM */
+#ifdef CONFIG_DISCONTIGMEM
+#define  __GFP_DISCONTIG 0x80
+#else /* CONFIG_DISCONTIGMEM */
+#define  __GFP_DISCONTIG 0
+#endif /* CONFIG_DISCONTIGMEM */
 
 #define GFP_BUFFER	(__GFP_WAIT)
 #define GFP_ATOMIC	(__GFP_HIGH)
--- mm/page_alloc.c	Tue Feb  1 14:25:03 2000
+++ mm/page_alloc.c	Wed Feb  2 18:17:57 2000
@@ -240,6 +240,14 @@
 
 	if (!freed && !(gfp_mask & __GFP_HIGH))
 		return 0;
+	/*
+	 * If discontig alloc case, caller should get back control,
+	 * to decide whether he wants to search a different node, or
+	 * he would rather stick to this node, possibly falling back
+	 * on lower zones. freed is set to 1 from above.
+	 */
+	if (gfp_mask & __GFP_DISCONTIG)
+		return 0;
 	return 1;
 }
 
@@ -467,6 +475,7 @@
 	for (i = 0; i < NR_GFPINDEX; i++) {
 		zonelist_t *zonelist;
 		zone_t *zone;
+		int nofallback = (i & __GFP_NOFALLBACK);
 
 		zonelist = pgdat->node_zonelists + i;
 		memset(zonelist, 0, sizeof(*zonelist));
@@ -493,14 +502,20 @@
 #endif
 					zonelist->zones[j++] = zone;
 				}
+				if (nofallback)
+					break;
 			case ZONE_NORMAL:
 				zone = pgdat->node_zones + ZONE_NORMAL;
 				if (zone->size)
 					zonelist->zones[j++] = zone;
+				if (nofallback)
+					break;
 			case ZONE_DMA:
 				zone = pgdat->node_zones + ZONE_DMA;
 				if (zone->size)
 					zonelist->zones[j++] = zone;
+				if (nofallback)
+					break;
 		}
 		zonelist->zones[j++] = NULL;
 	} 

