Merge branch 'toolbar' into img2img

Signed-off-by: Victor Seiji Hariki <victorseijih@gmail.com>

Former-commit-id: c8b0a3983dd07e64bc67d848814494be0ee8075d

README.md

@@ -1,6 +1,6 @@
 # hello there 🐠
 
-![openOutpaint creating some undersea wildlife](docs/01-demo-v3-c.gif)
+[openOutpaint creating some undersea wildlife](https://user-images.githubusercontent.com/1765167/203318284-a9f6970f-c9f1-44be-8c61-810aa5ed46be.webm)
 
 this is a completely vanilla javascript and html canvas outpainting convenience doodad built for the API optionally exposed by [AUTOMATIC1111's stable diffusion webUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui), operating similarly to a few others which certainly have superior functionality. this simply offers an alternative for my following vain desires:
 
@@ -23,13 +23,14 @@ this is a completely vanilla javascript and html canvas outpainting convenience
 - optional hi-res fix for blank/txt2img dreams which, if enabled, uses image width/height / 2 as firstpass size
 - import arbitrary images and superimpose on the canvas wherever you'd like ([extra fun with transparent .pngs!](#arbitrary_transparent))
 - "temporary" monitors at the bottom to see exactly what mask/image you're feeding img2img, no i'm certainly not using them as actual imagedata sources or anything
+- upscaler support for final output images _(NOTE: LDSR has had reports of not operating correctly when selected - please test and see if it works as expected)_
 - saves your preferences to browser localstorage for maximum convenience
 - undo/redo with ctrl+z/y keyboard shortcuts for additional maximum convenience
 
 ## collaborator credits 👑
 
 - [@jasonmhead](https://github.com/jasonmhead) - [the most minimal launch script](https://github.com/zero01101/openOutpaint/pull/1)
-- [@Kalekki](https://github.com/Kalekki) - [what i was calling "smart crop"](https://github.com/zero01101/openOutpaint/pull/2), [localstorage](https://github.com/zero01101/openOutpaint/pull/5), [right-click erase](https://github.com/zero01101/openOutpaint/pull/7), [delightful floating UI](https://github.com/zero01101/openOutpaint/pull/11), [mask erase fix](https://github.com/zero01101/openOutpaint/pull/17), [checkerboard background and non bonkers canvas borders](https://github.com/zero01101/openOutpaint/pull/24)
+- [@Kalekki](https://github.com/Kalekki) - [what i was calling "smart crop"](https://github.com/zero01101/openOutpaint/pull/2), [localstorage](https://github.com/zero01101/openOutpaint/pull/5), [right-click erase](https://github.com/zero01101/openOutpaint/pull/7), [delightful floating UI](https://github.com/zero01101/openOutpaint/pull/11), [mask erase fix](https://github.com/zero01101/openOutpaint/pull/17), [checkerboard background and non bonkers canvas borders](https://github.com/zero01101/openOutpaint/pull/24), [upscaling output image](https://github.com/zero01101/openOutpaint/pull/35)
 - [@seijihariki](https://github.com/seijihariki) - [realtime slider value updates, gracious code cleanup](https://github.com/zero01101/openOutpaint/pull/14), [blessed undo/redo](https://github.com/zero01101/openOutpaint/pull/21), [even more wildly massive rework of loads of my miserable of JS holy crap](https://github.com/zero01101/openOutpaint/pull/22), [undo/redo keyboard shortcuts and keyboard input support](https://github.com/zero01101/openOutpaint/pull/30), [scrumptious photography-shoppe-style history palette](https://github.com/zero01101/openOutpaint/commit/b12fc0d2a02074cb31c0ef35ce56d2bd02244908)
 - [@lifeh2o](https://www.reddit.com/user/lifeh2o/overview) - overmasking concept ~~that is still driving me crazy getting it to work right~~ ([a](https://www.reddit.com/r/StableDiffusion/comments/ywf8np/i_made_a_completely_local_offline_opensource/iwl6s06/),[b](https://www.reddit.com/r/StableDiffusion/comments/ys9lhq/kollai_an_infinite_multiuser_canvas_running_on/ivzygwk/?context=3)) [possible betterness?](https://github.com/zero01101/openOutpaint/commit/8002772ee6aa4b2f5b544af82cb6d545cf81368f)
 
@@ -96,7 +97,8 @@ you'll obviously need A1111's webUI installed before you can use this, thus you'
 - [x] image erase region in case you decide later that you're not too happy with earlier results (technically i guess you could just mask over the entire region you dislike but that's... bad)
 - [ ] controls for the rest of API-available options (e.g. ~~hires fix~~, inpaint fill modes, etc)
 - [x] ~~save user-set option values to browser localstorage to persist your preferred, uh, preferences~~
-- [ ] render progress spinner/bar
+- [x] render progress spinner/bar
+- [ ] make render progress bar prettier
 - [x] ~~smart crop downloaded image~~
 - [x] import external image and ~~scale/~~ superimpose at will on canvas for in/outpainting
 - [ ] scaling of imported arbitrary image before superimposition
@@ -167,6 +169,7 @@ imported a transparent clip of a [relatively famous happy lil kitty](https://com
 - 0.0.6 - absolutely brilliant undo/redo system, logical and straightforward enough to the point where even i can understand what it's doing [25681b3](https://github.com/zero01101/openOutpaint/commit/25681b3a83bbd7a1d1b3e675f26f141692d77c79)
 - 0.0.6.1 - finally think i've got overmasking working better with a bit of "humanization" to the automated masks, please play around with it and see if it's any better or just sucks in general [8002772](https://github.com/zero01101/openOutpaint/commit/8002772ee6aa4b2f5b544af82cb6d545cf81368f)
 - 0.0.6.5 - checkerboard background, far more attractive painted masking, HUGE code cleanup omg [74d5f13](https://github.com/zero01101/openOutpaint/commit/74d5f13aa582695e3e359ad46f7e629a25fb0091)
+- 0.0.6.9 - upscaler support for final output image [3b91a89](https://github.com/zero01101/openOutpaint/commit/3b91a89214e22930ad75fdc2d9e6e79a5f40ee82)
 
 ## what's with the fish?
 

docs/01-demo-v2.gif.REMOVED.git-id

@@ -1 +0,0 @@
-4b5fbd2adfcf11a8758f18d02ec6295942a3a941

docs/01-demo-v3-c.gif.REMOVED.git-id

@@ -1 +0,0 @@
-c3f0f5c0a67b0e6052db6d34d48f636fb06259d6

index.html

@@ -155,9 +155,19 @@ people, person, humans, human, divers, diver, glitch, error, text, watermark, ba
 				<button type="button" class="collapsible">Save/Load/New image</button>
 				<div class="content">
 					<label for="preloadImage">Load image:</label>
-					<input type="file" id="preloadImage" onchange="preloadImage()"
+					<input
-					accept="image/*" / style="width: 200px;"><br />
+						type="file"
+						id="preloadImage"
+						onchange="preloadImage()"
+						accept="image/*"
+						style="width: 200px" /><br />
 					<button onclick="downloadCanvas()">Save canvas</button><br />
+					<label for="upscalers">Choose upscaler</label>
+					<select id="upscalers"></select>
+					<button onclick="upscaleAndDownload()">
+						Upscale (might take a sec)</button
+					><br />
+
 					<button onclick="newImage()">Clear canvas</button>
 				</div>
 				<!-- Debug info -->

js/index.js

@@ -142,6 +142,7 @@ var arbitraryImageBase64; // seriously js cmon work with me here
 var placingArbitraryImage = false; // for when the user has loaded an existing image from their computer
 var marchOffset = 0;
 var stopMarching = null;
+var inProgress = false;
 var marchCoords = {};
 
 // info div, sometimes hidden
@@ -170,6 +171,7 @@ const bgCtx = bgCanvas.getContext("2d");
 function startup() {
 	checkIfWebuiIsRunning();
 	loadSettings();
+	getUpscalers();
 	drawBackground();
 	changeScaleFactor();
 	changeSampler();
@@ -211,21 +213,38 @@ function dream(
 	x,
 	y,
 	prompt,
-	extra = {method: endpoint, stopMarching: () => {}}
+	extra = {
+		method: endpoint,
+		stopMarching: () => {},
+		bb: {x, y, w: prompt.width, h: prompt.height},
+	}
 ) {
 	tmpImgXYWH.x = x;
 	tmpImgXYWH.y = y;
 	tmpImgXYWH.w = prompt.width;
 	tmpImgXYWH.h = prompt.height;
+	console.log(
+		"dreaming to " +
+			host +
+			url +
+			(extra.method || endpoint) +
+			":\r\n" +
+			JSON.stringify(prompt)
+	);
 	console.info(`dreaming "${prompt.prompt}"`);
 	console.debug(prompt);
-	postData(prompt, extra).then((data) => {
+
-		returnedImages = data.images;
+	// Start checking for progress
-		totalImagesReturned = data.images.length;
+	const progressCheck = checkProgress(extra.bb);
-		blockNewImages = true;
+	postData(prompt, extra)
-		//console.log(data); // JSON data parsed by `data.json()` call
+		.then((data) => {
-		imageAcceptReject(x, y, data, extra);
+			returnedImages = data.images;
-	});
+			totalImagesReturned = data.images.length;
+			blockNewImages = true;
+			//console.log(data); // JSON data parsed by `data.json()` call
+			imageAcceptReject(x, y, data, extra);
+		})
+		.finally(() => clearInterval(progressCheck));
 }
 
 async function postData(promptData, extra = null) {
@@ -251,6 +270,8 @@ async function postData(promptData, extra = null) {
 }
 
 function imageAcceptReject(x, y, data, extra = null) {
+	inProgress = false;
+	document.getElementById("progressDiv").remove();
 	const img = new Image();
 	img.onload = function () {
 		tempCtx.drawImage(img, x, y); //imgCtx for actual image, tmp for... holding?
@@ -262,7 +283,7 @@ function imageAcceptReject(x, y, data, extra = null) {
 		div.style.width = "200px";
 		div.style.height = "70px";
 		div.innerHTML =
-			'<button onclick="prevImg(this)">&lt;</button><button onclick="nextImg(this)">&gt;</button><span class="strokeText" id="currentImgIndex"></span><span class="strokeText"> of </span><span class="strokeText" id="totalImgIndex"></span><button onclick="accept(this)">Y</button><button onclick="reject(this)">N</button>';
+			'<button onclick="prevImg(this)">&lt;</button><button onclick="nextImg(this)">&gt;</button><span class="strokeText" id="currentImgIndex"></span><span class="strokeText"> of </span><span class="strokeText" id="totalImgIndex"></span><button onclick="accept(this)">Y</button><button onclick="reject(this)">N</button><span class="strokeText" id="estRemaining"></span>';
 		document.getElementById("tempDiv").appendChild(div);
 		document.getElementById("currentImgIndex").innerText = "1";
 		document.getElementById("totalImgIndex").innerText = totalImagesReturned;
@@ -410,6 +431,35 @@ function drawMarchingAnts(bb, offset) {
 	tgtCtx.strokeRect(bb.x, bb.y, bb.w, bb.h);
 }
 
+function checkProgress(bb) {
+	document.getElementById("progressDiv") &&
+		document.getElementById("progressDiv").remove();
+	// Skip image to stop using a ton of networking resources
+	endpoint = "progress?skip_current_image=true";
+	var div = document.createElement("div");
+	div.id = "progressDiv";
+	div.style.position = "absolute";
+	div.style.width = "200px";
+	div.style.height = "70px";
+	div.style.left = parseInt(bb.x + bb.w - 100) + "px";
+	div.style.top = parseInt(bb.y + bb.h) + "px";
+	div.innerHTML = '<span class="strokeText" id="estRemaining"></span>';
+	document.getElementById("tempDiv").appendChild(div);
+	return setInterval(() => {
+		fetch(host + url + endpoint)
+			.then((response) => response.json())
+			.then((data) => {
+				var estimate =
+					Math.round(data.progress * 100) +
+					"% :: " +
+					Math.floor(data.eta_relative) +
+					" sec.";
+
+				document.getElementById("estRemaining").innerText = estimate;
+			});
+	}, 500);
+}
+
 function mouseMove(evt) {
 	const rect = ovCanvas.getBoundingClientRect(); // not-quite pixel offset was driving me insane
 	const canvasOffsetX = rect.left;
@@ -643,8 +693,8 @@ function cropCanvas(sourceCanvas) {
 		return a - b;
 	});
 	var n = pix.x.length - 1;
-	w = pix.x[n] - pix.x[0];
+	w = pix.x[n] - pix.x[0] + 1;
-	h = pix.y[n] - pix.y[0];
+	h = pix.y[n] - pix.y[0] + 1;
 	// yup sure looks like it
 
 	try {
@@ -679,6 +729,144 @@ function checkIfWebuiIsRunning() {
 		});
 }
 
+function getUpscalers() {
+	/*
+	 so for some reason when upscalers request returns upscalers, the real-esrgan model names are incorrect, and need to be fetched from /sdapi/v1/realesrgan-models
+	 also the realesrgan models returned are not all correct, extra fun!
+	 LDSR seems to have problems so we dont add that either ->  RuntimeError: Number of dimensions of repeat dims can not be smaller than number of dimensions of tensor
+	 need to figure out why that is, if you dont get this error then you can add it back in
+
+	 Hacky way to get the correct list all in one go is to purposefully make an incorrect request, which then returns
+	 { detail: "Invalid upscaler, needs to be on of these: None , Lanczos , Nearest , LDSR , BSRGAN , R-ESRGAN General 4xV3 , R-ESRGAN 4x+ Anime6B , ScuNET , ScuNET PSNR , SwinIR_4x" }
+	 from which we can extract the correct list of upscalers
+	*/
+
+	// hacky way to get the correct list of upscalers
+	var upscalerSelect = document.getElementById("upscalers");
+	var extras_url =
+		document.getElementById("host").value + "/sdapi/v1/extra-single-image/"; // endpoint for upscaling, needed for the hacky way to get the correct list of upscalers
+	var empty_image = new Image(512, 512);
+	empty_image.src =
+		"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAFCAAAAABCAYAAAChpRsuAAAALklEQVR42u3BAQ0AAAgDoJvc6LeHAybtBgAAAAAAAAAAAAAAAAAAAAAAAAB47QD2wAJ/LnnqGgAAAABJRU5ErkJggg=="; //transparent pixel
+	var purposefully_incorrect_data = {
+		"resize-mode": 0, // 0 = just resize, 1 = crop and resize, 2 = resize and fill i assume based on theimg2img tabs options
+		upscaling_resize: 2,
+		upscaler_1: "fake_upscaler",
+		image: empty_image.src,
+	};
+
+	fetch(extras_url, {
+		method: "POST",
+		headers: {
+			Accept: "application/json",
+			"Content-Type": "application/json",
+		},
+		body: JSON.stringify(purposefully_incorrect_data),
+	})
+		.then((response) => response.json())
+		.then((data) => {
+			console.log("purposefully_incorrect_data response, ignore above error");
+			// result = purposefully_incorrect_data response: Invalid upscaler, needs to be on of these: None , Lanczos , Nearest , LDSR , BSRGAN , R-ESRGAN General 4xV3 , R-ESRGAN 4x+ Anime6B , ScuNET , ScuNET PSNR , SwinIR_4x
+			let upscalers = data.detail.split(": ")[1].trim().split(" , "); // converting the result to a list of upscalers
+			for (var i = 0; i < upscalers.length; i++) {
+				// if(upscalers[i] == "LDSR") continue; // Skip LDSR, see reason in the first comment // readded because worksonmymachine.jpg but leaving it here in case of, uh, future disaster?
+				var option = document.createElement("option");
+				option.text = upscalers[i];
+				option.value = upscalers[i];
+				upscalerSelect.add(option);
+			}
+		});
+
+	/* THE NON HACKY WAY THAT I SIMPLY COULD NOT GET TO PRODUCE A LIST WITHOUT NON WORKING UPSCALERS, FEEL FREE TO TRY AND FIGURE IT OUT
+
+	var url = document.getElementById("host").value + "/sdapi/v1/upscalers";
+	var realesrgan_url = document.getElementById("host").value + "/sdapi/v1/realesrgan-models";
+
+	// get upscalers
+	fetch(url)
+		.then((response) => response.json())
+		.then((data) => {
+			console.log(data);
+
+			for (var i = 0; i < data.length; i++) {
+				var option = document.createElement("option");
+
+				if (data[i].name.includes("ESRGAN") || data[i].name.includes("LDSR")) {
+					continue;
+				}
+				option.text = data[i].name;
+				upscalerSelect.add(option);
+			}
+		})
+		.catch((error) => {
+			alert(
+				"Error getting upscalers, please check console for additional info\n" +
+					error
+			);
+		});
+	// fetch realesrgan models separately
+	fetch(realesrgan_url)
+		.then((response) => response.json())
+		.then((data) => {
+			var model = data;
+			for(var i = 0; i < model.length; i++){
+				let option = document.createElement("option");
+				option.text = model[i].name;
+				option.value = model[i].name;
+				upscalerSelect.add(option);
+
+			}
+		
+	})
+	*/
+}
+
+async function upscaleAndDownload() {
+	// Future improvements: some upscalers take a while to upscale, so we should show a loading bar or something, also a slider for the upscale amount
+
+	// get cropped canvas, send it to upscaler, download result
+	var upscale_factor = 2; // TODO: make this a user input 1.x - 4.0 or something
+	var upscaler = document.getElementById("upscalers").value;
+	var croppedCanvas = cropCanvas(imgCanvas);
+	if (croppedCanvas != null) {
+		var upscaler = document.getElementById("upscalers").value;
+		var url =
+			document.getElementById("host").value + "/sdapi/v1/extra-single-image/";
+		var imgdata = croppedCanvas.toDataURL("image/png");
+		var data = {
+			"resize-mode": 0, // 0 = just resize, 1 = crop and resize, 2 = resize and fill i assume based on theimg2img tabs options
+			upscaling_resize: upscale_factor,
+			upscaler_1: upscaler,
+			image: imgdata,
+		};
+		console.log(data);
+		await fetch(url, {
+			method: "POST",
+			headers: {
+				Accept: "application/json",
+				"Content-Type": "application/json",
+			},
+			body: JSON.stringify(data),
+		})
+			.then((response) => response.json())
+			.then((data) => {
+				console.log(data);
+				var link = document.createElement("a");
+				link.download =
+					new Date()
+						.toISOString()
+						.slice(0, 19)
+						.replace("T", " ")
+						.replace(":", " ") +
+					" openOutpaint image upscaler_" +
+					upscaler +
+					".png";
+				link.href = "data:image/png;base64," + data["image"];
+				link.click();
+			});
+	}
+}
+
 function loadSettings() {
 	// set default values if not set
 	var _sampler =

js/ui/tool/dream.js

@@ -32,7 +32,7 @@ const dream_generate_callback = (evn, state) => {
 		// Use txt2img if canvas is blank
 		if (isCanvasBlank(bb.x, bb.y, bb.w, bb.h, imgCanvas)) {
 			// Dream
-			dream(bb.x, bb.y, request, {method: "txt2img"});
+			dream(bb.x, bb.y, request, {method: "txt2img", stopMarching, bb});
 		} else {
 			// Use img2img if not
 
@@ -71,7 +71,7 @@ const dream_generate_callback = (evn, state) => {
 			request.mask = auxCanvas.toDataURL();
 
 			// Dream
-			dream(bb.x, bb.y, request, {method: "img2img"});
+			dream(bb.x, bb.y, request, {method: "img2img", stopMarching, bb});
 		}
 	}
 };