0.0.5 - arbitrary image superimposition, etc

README.md

@@ -14,9 +14,11 @@ this is a completely vanilla javascript and html canvas outpainting convenience
  ## features
   - a big ol' 2560x1440 canvas for you to paint all over _(infinite canvas area planned, in //todo already)_
   - inpainting/touchup blob
-  - easily change samplers/steps/CFG/etc options for each "dream" summoned from the latent void 
+  - easily change samplers/steps/CFG/etc options for each dream summoned from the latent void 
   - optional grid snapping for precision
   - optional overmasking for better seams between outpaints (suggested by @lifeh2o ([a](https://www.reddit.com/r/StableDiffusion/comments/ywf8np/i_made_a_completely_local_offline_opensource/iwl6s06/),[b](https://www.reddit.com/r/StableDiffusion/comments/ys9lhq/kollai_an_infinite_multiuser_canvas_running_on/ivzygwk/?context=3)) and i think it's a slick idea)
+  - optional hi-res fix for blank/txt2img dreams which, if enabled, uses image width/height / 2 as firstpass size
+  - import arbitrary images and superimpose on the canvas wherever you'd like (extra fun with transparent .pngs!) ![imported images that have been _changed_](docs/03-arbimg.png)
   - "temporary" monitors at the bottom to see exactly what mask/image you're feeding img2img, no i'm certainly not using them as actual imagedata sources or anything
   - saves your preferences to browser localstorage for maximum convenience
 
@@ -29,6 +31,7 @@ this is a completely vanilla javascript and html canvas outpainting convenience
  -  technically you can run it directly in browser as a bare `file://` protocol webpage but that's _really_ not recommended as you'll have to add `null` as an accepted domain to your `--cors-allow-origins` option which just seems like it's a visibly poor decision 
  - a deliciously simple launch script (thanks [@jasonmhead](https://github.com/jasonmhead)! (https://github.com/zero01101/openOutpaint/pull/1)) is included to pop up a teensy tiny python-based local webserver, however you may have to manually `chmod +x openOutpaint.sh` on mac/linux 
  - the address http://127.0.0.1:3456 will be used as the host address for openOutpaint in the below quickstart; your local setup may use a different IP address or port.  you can of course modify the included launch script to point at a different port than 3456 if desired, as well
+ - if your scale factor is > 8 (generating an image larger than 512x512), try the "auto txt2img HR fix" option
 
 ### quickstart speedrun
 1. edit your `cors-allow-origins` to include https://zero01101.github.io and run webUI
@@ -45,15 +48,15 @@ this is a completely vanilla javascript and html canvas outpainting convenience
   - set your `Inpainting conditioning mask strength` to `1`
   - disable the `Apply color correction to img2img results to match original colors.` option (the last 2 options are found under the stable diffusion category in the settings tab by default unless you've already moved it to your quicksettings list, and if so, you know where to set them already)
  6. open your locally-hosted web server at http://127.0.0.1:3456 (or wherever, i'm not your boss)
- 7. update the host field if necessary to point at your stable diffusion API address, change my stupid prompts with whatever you want, click somewhere in the canvas, and wait
+ 7. update the host field if necessary to point at your stable diffusion API address, change my stupid prompts with whatever you want, click somewhere in the canvas, and wait _OR_ you can load an existing image from your computer using the "open img" button
  8. once an image appears*, click the `<` and `>` buttons at the bottom-left corner of the image to cycle through the others in the batch if you requested multiple (it defaults to 2 batch size, 2 batch count) - click `y` to choose one you like, or `n` to cancel that image generation batch outright and possibly try again
  9. now that you've got a starter, click somewhere near it to outpaint - try and include as much of the "context" as possible in the reticle for the best result convergence
  10. enable the mask mode to prepare previously rendered imagery for touchups/inpainting, then paint over the objectionable region; once your masked region is drawn, disable mask mode and change your prompt if necessary, then click over the canvas containing the mask you just painted to request the refined image(s)
  11. play around with the available options!  
   - scale factor affects the size of both the painting reticle and mask blob 
   - ...everything else is pretty much just a regular stable diffusion option so i presume you know how you use those
- 12. click "dl img" to save the cropped region of outpainted canvas (thanks [@Kalekki](https://github.com/Kalekki)! (https://github.com/zero01101/openOutpaint/pull/2))
+ 12. click "dl canvas" to save the cropped region of outpainted canvas (thanks [@Kalekki](https://github.com/Kalekki)! (https://github.com/zero01101/openOutpaint/pull/2))
- 13. refresh the page to start a blank canvas and start all over only to discover that it's like 2 AM and you have to go to sleep because you have work in about 4 hours 
+ 13. click "new image" to blank the canvas and start all over only to discover that it's like 2 AM and you have to go to sleep because you have work in about 4 hours 
 
 *if it _doesn't_ create an image, check your console output to see if you've got CORS errors 
 
@@ -61,19 +64,19 @@ this is a completely vanilla javascript and html canvas outpainting convenience
 ### in order of "priority"/likelihood of me doing it
 - [ ] lots and lots of readme updates (ongoing)
 - [ ] comment basically everything that isn't self documenting (ongoing)
-- [x] overmask seam of img2img 
+- [ ] CHORE: refactor all the duplicated JS code (ongoing, guaranteed to get worse before it gets better)
+- [x] overmask seam of img2img - BUG: it kinda sucks currently, just moves the seam instead of fixing it, i want to try to gradient-fade the edge but filter = 'blur(Ypx)' is _awful_ for this and my remedial per-pixel loops crash the browser because i am the embodiment of inefficiency
 - [x] split out CSS to its own file (remedial cleanup task)
-- [ ] ability to blank/new canvas without making the user refresh the page because that's pretty janky
+- [x] ability to blank/new canvas without making the user refresh the page because that's pretty janky
 - [ ] add error handling for async/XHR POST in case of, yknow, errors
 - [ ] image erase region in case you decide later that you're not too happy with earlier results (technically i guess you could just mask over the entire region you dislike but that's... bad)
-- [ ] controls for the rest of API-available options (e.g. hires fix, inpaint fill modes, etc)
+- [ ] controls for the rest of API-available options (e.g. ~~hires fix~~, inpaint fill modes, etc)
 - [ ] ~~save user-set option values to browser localstorage to persist your preferred, uh, preferences~~ (thanks again [@Kalekki](https://github.com/Kalekki)! (https://github.com/zero01101/openOutpaint/pull/5))
 - [ ] render progress spinner/bar
 - [ ] ~~smart crop downloaded image~~ 
-- [ ] import external image and scale/superimpose at will on canvas for in/outpainting
+- [x] import external image and ~~scale/~~superimpose at will on canvas for in/outpainting
+- [ ] scaling of imported arbitrary image before superimposition
 - [ ] "numpad" selector for determining how reticle is anchored against actual mouse cursor (currently works like a "5" [center] on the "numpad" paradigm)
-- [ ] BUG: figure out where that stupid 1-pixel offset is happening between approve/reject state and committing to an image, it doesn't affect output but it's _super_ obnoxious  
-- [ ] BUG: make erase mask actually work, enable the control if you dare
 - [ ] discrete size control for mask and target reticle, discrete x/y axes for reticle
 - [ ] floating/togglable menu leftnav bar with categorized/sensibly laid-out options
 - [ ] infinite canvas
@@ -93,6 +96,13 @@ i am begging you, yes you personally reading this, please fix my horrible code a
 ## bug reports
 please do! kindly indicate your OS, browser, versions of both, any errors in devtools/console output, what you were trying to do, what you expected, what happened unexpectedly or incorrectly, if something caught fire (please call the fire department first), the usual
 
+## known bugs :(
+ - generated images display +1px on x/y during approve/reject state, doesn't affect output, just annoying
+ - erase mask is like entirely broken
+ - odd-numbered scale factors don't snap correctly
+ - arbitrary "pasted" images require clicking twice to place them and i _don't know why_ [(yes i do)](#terrible), just getting them to be arbitrarily placable was a giant pain because i'm not got the smarts
+ - selecting an aribtrary image by double-clicking it in the file picker can sometimes trigger a dream request that errors out if your file picker is "above" the canvas; i tried to alleviate that by temporarily removing the mouse(move/down/up) handlers for the canvas context on selection of a file, but i'm POSITIVE it's an improper solution and not quite sure if it's even fully effective
+
 ## warranty
 [lmao](https://github.com/moyix/fauxpilot#support-and-warranty)
 
@@ -109,4 +119,5 @@ _(see https://github.com/zero01101/openOutpaint/commit/92ab9d231542ea5f7a3c85563
 - 0.0.4 - batch size/batch count, approve/reject system implementations, snap-to-grid, other people are now allowed to see this thing [01f8c6a](https://github.com/zero01101/openOutpaint/commit/01f8c6ab3f49739439a0990d6f5f0967a9a0bf12)
 - 0.0.4.1 - extremely minor revisions [02cb01a](https://github.com/zero01101/openOutpaint/commit/02cb01ac062ef93878ff4161eabcedfa8e125be6)
 - 0.0.4.2 - pull requests (&lt;3), downloaded images now have a timestamped name, css breakout because hopefully this will become halfway attractive enough to benefit from non-inline stylesheets [70ad4fe](https://github.com/zero01101/openOutpaint/commit/70ad4fe081bdbd507afc5af3cc2a4435924b66e3)
-- 0.0.4.3 - overmasking [fca2e01](https://github.com/zero01101/openOutpaint/commit/fca2e01b8a4ecfe3d062c4090d5886e1033e8f38)
+- 0.0.4.3 - overmasking, settings saved to localstorage [fca2e01](https://github.com/zero01101/openOutpaint/commit/fca2e01b8a4ecfe3d062c4090d5886e1033e8f38)
+- 0.0.5 - import arbitrary image from user's machine, "auto" txt2img hires fix, Very Important "new image" button 

index.html

@@ -3,8 +3,9 @@
 
 <head>
     <meta charset="utf-8" />
-    <title>openOutpaint 0.0.4.3</title>
+    <title>openOutpaint 0.0.5</title>
     <link href="css/index.css" rel="stylesheet" />
+    <link rel="icon" type="image/x-icon" href="favicon.ico">
 </head>
 
 <body>
@@ -31,6 +32,7 @@
                     <span id="snapY"></span>
                     <br />
                 </div>
+                <button onclick="newImage()">new image</button><br />
                 <label for="host">host:</label>
                 <input id="host" value="http://127.0.0.1:7860"><br />
                 <label for="scaleFactor">scale factor:</label>
@@ -38,8 +40,7 @@
                 <input type="range" id="scaleFactor" name="scaleFactor" min="1" max="16" value="8"
                     onchange="changeScaleFactor()" /><br />
                 <label for="prompt">prompt:</label>
-                <textarea
+                <textarea id="prompt">oceanographic study, underwater wildlife, award winning</textarea><br />
-                    id="prompt">oceanographic study, underwater wildlife, award winning documentary footage screenshot</textarea><br />
                 <label for="negPrompt">negative prompt:</label>
                 <textarea
                     id="negPrompt">people, person, humans, human, divers, diver, glitch, error, text, watermark, bad quality, blurry</textarea><br />
@@ -52,6 +53,8 @@
                 <input type="checkbox" id="cbxPaint" onchange="changePaintMode()"><br />
                 <label for="cbxErase"><s>erase mask?</s></label>
                 <input type="checkbox" id="cbxErase" onchange="changeEraseMode()" disabled="disabled"><br />
+                <label for="cbxHRFix">auto txt2img HRfix?</label>
+                <input type="checkbox" id="cbxHRFix" onchange="changeHiResFix()"><br />
                 <label for="cbxOverMask">overmasking?</label>
                 <input type="checkbox" id="cbxOverMask" onchange="changeOverMask()" checked="checked"><br />
                 <label for="overMaskPx">overmask px:</label>
@@ -125,7 +128,9 @@
                 <span id="maskBlurText"></span><br />
                 <input type="number" id="maskBlur" name="maskBlur" min="0" max="64" value="0" step="1"
                     onchange="changeMaskBlur()" /><br />
-                <button onclick="downloadImage()">dl img</button>
+                <label for="preloadImage">load local image:</label>
+                <input type="file" id="preloadImage" onchange="preloadImage()" accept="image/*" /><br />
+                <button onclick="downloadCanvas()">dl canvas</button>
             </div>
             <div id="canvasHolder" class="canvasHolder">
                 <canvas id="backgroundCanvas" class="mainCanvases backgroundCanvas" width="2560" height="1440"
@@ -138,7 +143,7 @@
                     <p>lol ur browser sucks</p>
                 </canvas>
                 <canvas id="tempCanvas" class="mainCanvases tempCanvas" width="2560" height="1440" style="z-index: 2;">
-                    <!-- temporary canvas on which images being selected/rejected are superimposed -->
+                    <!-- temporary canvas on which images being selected/rejected or imported arbitrary images are superimposed -->
                     <p>lol ur browser sucks</p>
                 </canvas>
                 <canvas id="targetCanvas" class="mainCanvases targetCanvas" width="2560" height="1440"

js/index.js

@@ -20,6 +20,9 @@ var stableDiffusionData = { //includes img2img data but works for txt2img just f
     init_images: [],
     inpaint_full_res: false,
     inpainting_fill: 2,
+    enable_hr: false,
+    firstphase_width: 0,
+    firstphase_height: 0,
     // here's some more fields that might be useful
 
     // ---txt2img specific:
@@ -79,6 +82,12 @@ var totalImagesReturned;
 var overMask = true;
 var overMaskPx = 10;
 var drawTargets = []; // is this needed?  i only draw the last one anyway...
+var dropTargets = []; // uhhh yeah similar to the above but for arbitrary dropped images
+var arbitraryImage;
+var arbitraryImageData;
+var arbitraryImageBitmap;
+var arbitraryImageBase64; // seriously js cmon work with me here
+var placingArbitraryImage = false; // for when the user has loaded an existing image from their computer
 
 // info div, sometimes hidden
 let mouseXInfo = document.getElementById("mouseX");
@@ -118,12 +127,28 @@ function startup() {
     changeSeed();
     changeOverMask();
     changeOverMaskPx();
+    changeHiResFix();
     document.getElementById("overlayCanvas").onmousemove = mouseMove;
     document.getElementById("overlayCanvas").onmousedown = mouseDown;
     document.getElementById("overlayCanvas").onmouseup = mouseUp;
     document.getElementById("scaleFactor").value = scaleFactor;
 }
 
+function drop(imageParams) {
+    const img = new Image();
+    img.src = arbitraryImageBase64;
+    if (img.complete) {
+        writeArbitraryImage(img, imageParams.x, imageParams.y)
+    }
+}
+
+function writeArbitraryImage(img, x, y) {
+    imgCtx.drawImage(img, x, y);
+    blockNewImages = false;
+    placingArbitraryImage = false;
+    document.getElementById("preloadImage").files = null;
+}
+
 function dream(x, y, prompt) {
     tmpImgXYWH.x = x;
     tmpImgXYWH.y = y;
@@ -178,7 +203,7 @@ function imageAcceptReject(x, y, data) {
     // set the image displayed as the first regardless of batch size/count
     imageIndex = 0;
     // load the image data after defining the closure
-    img.src = "data:image/png;base64," + returnedImages[imageIndex];  //TODO need way to dream batches and select from results
+    img.src = "data:image/png;base64," + returnedImages[imageIndex];
 }
 
 function accept(evt) {
@@ -199,6 +224,12 @@ function reject(evt) {
     blockNewImages = false;
 }
 
+function newImage(evt) {
+    clearBackupMask();
+    clearTargetMask();
+    clearImgMask();
+}
+
 function prevImg(evt) {
     if (imageIndex == 0) {
         imageIndex = totalImagesReturned;
@@ -256,6 +287,10 @@ function clearTargetMask() {
     tgtCtx.clearRect(0, 0, tgtCanvas.width, tgtCanvas.height);
 }
 
+function clearImgMask() {
+    imgCtx.clearRect(0, 0, imgCanvas.width, imgCanvas.height);
+}
+
 function placeImage() {
     const img = new Image();
     img.onload = function () {
@@ -278,7 +313,7 @@ function snap(i) {
     var scaleOffset = 0;
     if (scaleFactor % 2 != 0) {
         // odd number, snaps to center of cell, oops
-        scaleOffset = 32;
+        scaleOffset = (basePixelCount / 2);
     }
     var snapOffset = i % basePixelCount - scaleOffset;
     if (snapOffset == 0) {
@@ -299,7 +334,18 @@ function mouseMove(evt) {
     snapXInfo.innerText = canvasX + snap(canvasX);
     snapYInfo.innerText = canvasY + snap(canvasY);
     ovCtx.clearRect(0, 0, ovCanvas.width, ovCanvas.height); // clear out the previous mouse cursor 
-    if (!paintMode) {
+    if (placingArbitraryImage) {
+        // ugh refactor so this isn't duplicated between arbitrary image and dream reticle modes
+        snapOffsetX = 0;
+        snapOffsetY = 0;
+        if (snapToGrid) {
+            snapOffsetX = snap(canvasX);
+            snapOffsetY = snap(canvasY);
+        }
+        finalX = snapOffsetX + canvasX;
+        finalY = snapOffsetY + canvasY;
+        ovCtx.drawImage(arbitraryImage, parseInt(finalX - ((basePixelCount * scaleFactor) / 2)), parseInt(finalY - ((basePixelCount * scaleFactor) / 2)));
+    } else if (!paintMode) {
         // draw targeting square reticle thingy cursor
         ovCtx.strokeStyle = "#00000077";
         snapOffsetX = 0;
@@ -344,15 +390,23 @@ function mouseMove(evt) {
 }
 
 function mouseDown(evt) {
-    if (paintMode) {
+    const rect = ovCanvas.getBoundingClientRect()
-        const rect = ovCanvas.getBoundingClientRect() // not-quite pixel offset was driving me insane
+    if (placingArbitraryImage) {
+        var nextBox = {};
+        nextBox.x = evt.clientX - ((basePixelCount * scaleFactor) / 2) - rect.left; //origin is middle of the frame 
+        nextBox.y = evt.clientY - ((basePixelCount * scaleFactor) / 2) - rect.top; //TODO make a way to set the origin to numpad dirs?
+        nextBox.w = basePixelCount * scaleFactor;
+        nextBox.h = basePixelCount * scaleFactor;
+        dropTargets.push(nextBox);
+    } else if (paintMode) {
+        //const rect = ovCanvas.getBoundingClientRect() // not-quite pixel offset was driving me insane
         const canvasOffsetX = rect.left;
         const canvasOffsetY = rect.top;
         prevMouseX = mouseX = evt.clientX - canvasOffsetX;
         prevMouseY = mouseY = evt.clientY - canvasOffsetY;
         clicked = true;
     } else {
-        const rect = ovCanvas.getBoundingClientRect()
+        //const rect = ovCanvas.getBoundingClientRect()
         var nextBox = {};
         nextBox.x = evt.clientX - ((basePixelCount * scaleFactor) / 2) - rect.left; //origin is middle of the frame 
         nextBox.y = evt.clientY - ((basePixelCount * scaleFactor) / 2) - rect.top; //TODO make a way to set the origin to numpad dirs?
@@ -363,7 +417,26 @@ function mouseDown(evt) {
 }
 
 function mouseUp(evt) {
-    if (paintMode) {
+    if (placingArbitraryImage) {
+        //TODO DO SOMETHING
+        // jeez i REALLY need to refactor tons of this to not be duplicated all over, that's definitely my next chore after figuring out that razza frazza overmask fade
+        var target = dropTargets[dropTargets.length - 1]; //get the last one... why am i storing all of them?
+        snapOffsetX = 0;
+        snapOffsetY = 0;
+        if (snapToGrid) {
+            snapOffsetX = snap(target.x);
+            snapOffsetY = snap(target.y);
+        }
+        finalX = snapOffsetX + target.x;
+        finalY = snapOffsetY + target.y;
+
+        drawThis.x = finalX;
+        drawThis.y = finalY;
+        drawThis.w = target.w;
+        drawThis.h = target.h;
+        drawIt = drawThis; // i still think this is really stupid and redundant and unnecessary and redundant
+        drop(drawIt);
+    } else if (paintMode) {
         clicked = false;
         return;
     } else {
@@ -486,6 +559,10 @@ function mouseUp(evt) {
                         // but it is crushingly inefficient i'm sure
                         // BUT IT WORKS and i came up with it all by myself because i'm a big boy
 
+                        // ... sigh it doesn't work _well_
+                        // just moves the seam 
+                        //TODO find a way to fade/gradient the edge without making weird artifacts or literally crashing the browser with inefficient data storage
+
                         // west
                         var potentialPixelIndex = ((currentMaskPixelY * drawIt.w + currentMaskPixelX) * 4) - (j * 4);
                         var potentialPixelX = (potentialPixelIndex / 4) % drawIt.w;
@@ -543,7 +620,6 @@ function mouseUp(evt) {
                             }
                         }
                     }
-
                 }
 
                 // also check for painted masks in region, add them as white pixels to mask canvas
@@ -595,6 +671,8 @@ function mouseUp(evt) {
             stableDiffusionData.negative_prompt = document.getElementById("negPrompt").value;
             stableDiffusionData.width = drawIt.w;
             stableDiffusionData.height = drawIt.h;
+            stableDiffusionData.firstphase_height = (drawIt.h / 2);
+            stableDiffusionData.firstphase_width = (drawIt.w / 2);
             dream(drawIt.x, drawIt.y, stableDiffusionData);
         }
     }
@@ -664,6 +742,11 @@ function changeOverMaskPx() {
     overMaskPx = document.getElementById("overMaskPx").value;
 }
 
+function changeHiResFix() {
+    stableDiffusionData.enable_hr = Boolean(document.getElementById("cbxHRFix").checked);
+    localStorage.setItem("enable_hr", stableDiffusionData.enable_hr);
+}
+
 function isCanvasBlank(x, y, w, h, specifiedCanvas) {
     var canvas = document.getElementById(specifiedCanvas.id);
     return !canvas.getContext('2d')
@@ -687,7 +770,38 @@ function drawBackground() {
     }
 }
 
-function downloadImage() {
+function preloadImage() {
+    // gonna legit scream 
+    document.getElementById("overlayCanvas").onmousemove = null;
+    document.getElementById("overlayCanvas").onmousedown = null;
+    document.getElementById("overlayCanvas").onmouseup = null;
+
+    var file = document.getElementById("preloadImage").files[0];
+    var reader = new FileReader();
+    reader.onload = function (evt) {
+        var imgCanvas = document.createElement("canvas");
+        var imgCtx = imgCanvas.getContext("2d");
+        arbitraryImage = new Image();
+        arbitraryImage.onload = function () {
+            blockNewImages = true;
+            // now put it into imagedata for canvas fun
+            imgCanvas.width = arbitraryImage.width;
+            imgCanvas.height = arbitraryImage.height;
+            imgCtx.drawImage(arbitraryImage, 0, 0);
+            arbitraryImageData = imgCtx.getImageData(0, 0, arbitraryImage.width, arbitraryImage.height); // can't use that to drawImage on a canvas...
+            arbitraryImageBitmap = createImageBitmap(arbitraryImageData); // apparently that either... maybe just the raw image?
+            arbitraryImageBase64 = imgCanvas.toDataURL();
+            placingArbitraryImage = true;
+            document.getElementById("overlayCanvas").onmousemove = mouseMove;
+            document.getElementById("overlayCanvas").onmousedown = mouseDown;
+            document.getElementById("overlayCanvas").onmouseup = mouseUp;
+        }
+        arbitraryImage.src = evt.target.result;
+    }
+    reader.readAsDataURL(file);
+}
+
+function downloadCanvas() {
     var link = document.createElement('a');
     link.download = new Date().toISOString().slice(0, 19).replace('T', ' ').replace(':', ' ') + ' openOutpaint image.png';
     var croppedCanvas = cropCanvas(imgCanvas);
@@ -748,6 +862,7 @@ function loadSettings() {
     var _scaleFactor = localStorage.getItem("scaleFactor") == null ? 8 : localStorage.getItem("scaleFactor");
     var _mask_blur = localStorage.getItem("mask_blur") == null ? 0 : localStorage.getItem("mask_blur");
     var _seed = localStorage.getItem("seed") == null ? -1 : localStorage.getItem("seed");
+    var _enable_hr = Boolean(localStorage.getItem("enable_hr") == (null || "false") ? false : localStorage.getItem("enable_hr"));
 
     // set the values into the UI
     document.getElementById("samplerSelect").value = String(_sampler);
@@ -758,4 +873,5 @@ function loadSettings() {
     document.getElementById("scaleFactor").value = Number(_scaleFactor);
     document.getElementById("maskBlur").value = Number(_mask_blur);
     document.getElementById("seed").value = Number(_seed);
+    document.getElementById("cbxHRFix").checked = Boolean(_enable_hr);
 }