We need a feature for sending a raw/binary post request.

[Baltazar500]

Hi.

The inability to send raw/binary forces to use curl to generate POST requests (which for example contain json '{"aaa": "bbb"}' and cannot be sent in urlencode format).

We need a feature for sending a raw/binary post request.

[Reino17]

Have you seen https://stackoverflow.com/a/69599480/2703456 ?

[Baltazar500]

Have you seen https://stackoverflow.com/a/69599480/2703456 ?

No, I have not seen. But is this case considered there (json is sent and extracted) ?

xidel -se '($json)().ghi' --post-data='{"abc":"def"}'

Can you give an example?

[Reino17]

I'm sorry, but I don't understand what you mean. I'm beginning to doubt whether I even correctly understood your question in the first place.

Maybe you could give a working curl command instead which you want xidel to do.

[Baltazar500]

Maybe you could give a working curl command instead which you want xidel to do.

curl -skL --data-binary '{"abc":"def"}' http://example.com

[Reino17]

This is not a working example, but anyway... if you want xidel to POST a JSON-string (with --post / -d), then you have to surround it with { }, because -d uses xidel's extended strings. So that would be: -d '{{"abc":"def"}}'.

[Baltazar500]

This is not a working example, but anyway... if you want xidel to POST a JSON-string (with --post / -d), then you have to surround it with {``} because -d uses xidel's extended strings. So that would be: -d '{{"abc":"def"}}'.

Oh. Thanks. I duplicated all the brackets and the request via xidel was sent successfully. Should I close the issue? Or not ? I would like to do without duplication and send data as is :)

[Reino17]

http://videlibri.sourceforge.net/xidel_readme.txt:

Extended strings: x"..{..}.."

If a string is prefixed by an "x", all expressions inside {}-parentheses
are evaluated, like in the value of a direct attribute constructor.
E.g. x"There are {1+2+3} elements" prints "There are 6 elements".

(The following commands starting with $ are for Linux and those starting with C:\> for Windows)

$ xidel -se 'let $a:={"abc":"def"} return concat($a/abc,"ghi")'
C:\>xidel -se "let $a:={'abc':'def'} return concat($a/abc,'ghi')"
defghi

$ xidel -se 'let $a:={"abc":"def"} return $a/abc||"ghi"'
C:\>xidel -se "let $a:={'abc':'def'} return $a/abc||'ghi'"
defghi

$ xidel -se 'let $a:={"abc":"def"} return x"{$a/abc}ghi"'
C:\>xidel -se "let $a:={'abc':'def'} return x'{$a/abc}ghi'"
defghi

Three ways to concat strings, of which the 3rd is using an "extended string".

$ xidel -se '{"abc":"def"}'
C:\>xidel -se "{'abc':'def'}"
{
  "abc": "def"      # This is JSON
}

$ xidel -se '"{""abc"":""def""}"'
C:\>xidel -se "'{\"abc\":\"def\"}'"
{"abc":"def"}       # This is a string(!)

In an extended string the parentheses need to be escaped:

$ xidel -se 'x"{{""abc"":""def""}}"'
C:\>xidel -se "x'{{\"abc\":\"def\"}}'"
{"abc":"def"}

For a simple JSON like this it's not such a big problem, but imagine this more complex JSON:

$ xidel -se 'x"{{""a"":1,""b"":{{""x"":""y""}},""c"":[{{""z"":null}}]}}"'
C:\>xidel -se "x'{{\"a\":1,\"b\":{{\"x\":\"y\"}},\"c\":[{{\"z\":null}}]}}'"
{"a":1,"b":{"x":"y"},"c":[{"z":null}]}

This is not fun anymore, having to manually escape all the parentheses. Luckily there's serialize():

$ xidel -se 'x"{serialize({"a":1,"b":{"x":"y"},"c":[{"z":null}]},{"method":"json"})}"'
C:\>xidel -se "x'{serialize({'a':1,'b':{'x':'y'},'c':[{'z':null}]},{'method':'json'})}'"
{"a":1,"b":{"x":"y"},"c":[{"z":null}]}

In an extended string you obviously still have to surround it with parentheses ({ serialize(...) }), but then you can enter the JSON as-is.

Now, the string entered for --post / -d is always an extended string(!), so all parentheses to be escaped:

$ xidel -s -d '{{"abc":"def"}}' "http://videlibri.sourceforge.net/xidelecho.php" -e '$raw'
C:\>xidel -s -d "{{\"abc\":\"def\"}}" "http://videlibri.sourceforge.net/xidelecho.php" -e "$raw"
<xml>
<meth>POST</meth>
<raw>{"abc":"def"}</raw>
</xml>

$ xidel -s -d '{{"a":1,"b":{{"x":"y"}},"c":[{{"z":null}}]}}' "http://videlibri.sourceforge.net/xidelecho.php" -e '$raw'
C:\>xidel -s -d "{{\"a\":1,\"b\":{{\"x\":\"y\"}},\"c\":[{{\"z\":null}}]}}" "http://videlibri.sourceforge.net/xidelecho.php" -e "$raw"
<xml>
<meth>POST</meth>
<raw>{"a":1,"b":{"x":"y"},"c":[{"z":null}]}</raw>
</xml>

But again, better to use serialize() for complex JSONs:

$ xidel -s -d '{serialize({"a":1,"b":{"x":"y"},"c":[{"z":null}]},{"method":"json"})}' "http://videlibri.sourceforge.net/xidelecho.php" -e '$raw'
C:\>xidel -s -d "{serialize({'a':1,'b':{'x':'y'},'c':[{'z':null}]},{'method':'json'})}" "http://videlibri.sourceforge.net/xidelecho.php" -e "$raw"
<xml>
<meth>POST</meth>
<raw>{"a":1,"b":{"x":"y"},"c":[{"z":null}]}</raw>
</xml>

[Reino17]

You may also find https://stackoverflow.com/a/60890855/2703456 useful.

[Reino17]

Should I close the issue?

No feedback, so yes, I believe this issue can be closed.

[Baltazar500]

No feedback, so yes, I believe this issue can be closed.

@Reino17, Thanks for the examples. I will use the serialize() variant

[Baltazar500]

I would like to reopen an issue for a "similar" reason. A binary POST data upload is needed to use it with file hosting/image hosting/etc. This will help upload files without using CURL and parse the page to extract the URL after uploading :)

[benibela]

Raw/binary/file post are different things

• Raw is to not support some special characters in the parameter

• Binary is to not change the data. Curl has non-binary mode where it normalizes line endings and a binary mode where it does not normalize line endings. Xidel never changes the line endings on -d afair, so it is kind of always binary.

• You can submit the content of a file with -d '{file:read-text("filename")}'
  Curl has -d @filename to load file content. I could implement that, but perhaps that breaks existing scripts. Xidel does not need all the request options of curl, because XPath can be used to build any request.
  Although file:read-text might be a bad idea for non-text. (it does not check if the input is text, but if I would follow the standard precisely, it would raise an exception "invalid input" on non-utf8 files.)
  -d '{file:read-binary("filename")}' would be better, but currently it does not work, because it would send everything base64 encoded.

• File uploading is different. Usually websites uses multipart encoding for file upload. It MIME encodes the file and includes the filename. I have implemented the curl syntax for multipart file uploading:
  -F 'parametername=@/tmp/filename'
  (do not confuse --form/-F with --follow/-f)

• Then Xidel can submit HTML forms. There are special options to upload a file when submitting such a form: http://sitewiththeform -f 'form(//form, {"parametername": { "file": "/tmp/filename" } } )' (and there are more options)

C:>xidel -s -d "{serialize({'a':1,'b':{'x':'y'},'c':[{'z':null}]},{'method':'json'})}" "http://videlibri.sourceforge.net/xidelecho.php" -e "$raw"

Perhaps I change it, so you can skip the serialize function, e.g. xidel -s -d "{({'a':1,'b':{'x':'y'},'c':[{'z':null}]})}" "http://videlibri.sourceforge.net/xidelecho.php" -e "$raw"

But I cannot decide whether an object like {"a": "b", "c": "d"} should be send as JSON or url-encoded as a=b&c=d ?

And if it was sent as JSON, someone would try it with escapes like {"a": "b\\", "c": "d"} and then being confused that it sends {"a": "b\\\\", "c": "d"} because "b\\" is read as XPath string rather than a JSON string

[Baltazar500]

@benibela, thanks for your reply ;)

File uploading is different. Usually websites uses multipart encoding for file upload. It MIME encodes the file and includes the filename.

This is my case :)

I have implemented the curl syntax for multipart file uploading:

Does not work. GET is sent instead of POST. The header "Content-Type: multipart / form-data" doesn't help in any way either.

method=POST + header "Content-Type: multipart/form-data"

xidel-0.9.9.20211123.8232.023d1f1f656e.linux32 --user-agent='Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:52.0) Gecko/20100101 Firefox/52.0' --method=POST http://sendpic.org/upload.php --header "Content-Type: multipart/form-data" -F "img[]=@./ScreenShot_0009.jpg" -F "type=local_pic" -F "orig_img_a=orig_do_not_change" -F "orig_q_jpeg=85" -F "orig_q_png=6" -F "create_preview=1" -F "thumb_q=85" -F "thumb_img=thumb_w" -F "thumb_w=195" -F "thumb_txt=thumb_txt_size" -se "/html/body/center/table/tbody/tr[2]/td/fieldset/fieldset[3]/input/@value"

header "Content-Type: multipart/form-data"

xidel-0.9.9.20211123.8232.023d1f1f656e.linux32 --user-agent='Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:52.0) Gecko/20100101 Firefox/52.0' http://sendpic.org/upload.php --header "Content-Type: multipart/form-data" -F "img[]=@./ScreenShot_0009.jpg" -F "type=local_pic" -F "orig_img_a=orig_do_not_change" -F "orig_q_jpeg=85" -F "orig_q_png=6" -F "create_preview=1" -F "thumb_q=85" -F "thumb_img=thumb_w" -F "thumb_w=195" -F "thumb_txt=thumb_txt_size" -se "/html/body/center/table/tbody/tr[2]/td/fieldset/fieldset[3]/input/@value"

Without method=POST + header "Content-Type: multipart/form-data"

xidel-0.9.9.20211123.8232.023d1f1f656e.linux32 --user-agent='Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:52.0) Gecko/20100101 Firefox/52.0' http://sendpic.org/upload.php -F "img[]=@./ScreenShot_0009.jpg" -F "type=local_pic" -F "orig_img_a=orig_do_not_change" -F "orig_q_jpeg=85" -F "orig_q_png=6" -F "create_preview=1" -F "thumb_q=85" -F "thumb_img=thumb_w" -F "thumb_w=195" -F "thumb_txt=thumb_txt_size" -se "/html/body/center/table/tbody/tr[2]/td/fieldset/fieldset[3]/input/@value"

Everything is successfully sent via curl with the same parameters. :\

[benibela]

You need to put the options before the url

[Baltazar500]

You need to put the options before the url

Thanks, it works.

It's crazy :), but the request passed with all parameters. Can you fix this in the next builds ?

[Reino17]

Perhaps I change it, so you can skip the serialize function, e.g. xidel -s -d "{({'a':1,'b':{'x':'y'},'c':[{'z':null}]})}" "http://videlibri.sourceforge.net/xidelecho.php" -e "$raw"

I would vote against. The fact that --post/-d behaves differently than -e in the sense that it always uses extended strings is already enough inconsistency as far as I'm concerned. More different rules/behavior would be really confusing.

Thanks, it works.

$ xidel -s \
  -F "img[][email protected]" \
  -F "type=local_pic" \
  -F "orig_img_a=orig_do_not_change" \
  -F "orig_q_jpeg=85" \
  -F "orig_q_png=6" \
  -F "create_preview=1" \
  -F "thumb_q=85" \
  -F "thumb_img=thumb_w" \
  -F "thumb_w=195" \
  -F "thumb_txt=thumb_txt_size" \
  http://sendpic.org/upload.php \
  -e '//fieldset[3]/input/@value'

Never used this feature before, but this works for me too.
When I put everything on one line I'm getting:

$ xidel -s -F "img[][email protected];type=local_pic;orig_img_a=orig_do_not_change;orig_q_jpeg=85;orig_q_png=6;create_preview=1;thumb_q=85;thumb_img=thumb_w;thumb_w=195;thumb_txt=thumb_txt_size" http://sendpic.org/upload.php -e '//fieldset[3]/input/@value'
An unhandled exception occurred at $00432A20:
EXidelException: Unknown option in img[][email protected];type=local_pic;orig_img_a=orig_do_not_change;orig_q_jpeg=85;orig_q_png=6;create_preview=1;thumb_q=85;thumb_img=thumb_w;thumb_w=195;thumb_txt=thumb_txt_size

This is supposed to happen? And how would form() exactly work in this case? This at least doesn't:

$ xidel -s http://sendpic.org \
  -f 'form(
    //form,
    {
      "img[]":"@download.gif",
      "type":"local_pic",
      "orig_img_a":"orig_do_not_change",
      "orig_q_jpeg":85,
      "orig_q_png":6,
      "create_preview":1,
      "thumb_q":85,
      "thumb_img":"thumb_w",
      "thumb_w":195,
      "thumb_txt":"thumb_txt_size"
    }
  )' \
  -e '//fieldset[3]/input/@value'

It's crazy :), but the request passed with all parameters. Can you fix this in the next builds ?

Maybe benibela understands what you mean, but I don't. So I'm curious, what exactly do you want fixed?

Btw, a possible coloring bug?

[benibela]

, but the request passed with all parameters. Can you fix this in the next builds ?

Xidel always expects HTTP/upload options before the url

I would vote against. The fact that --post/-d behaves differently than -e in the sense that it always uses extended strings is already enough inconsistency as far as I'm concerned. More different rules/behavior would be really confusing.

I could change it for all extended strings

At least for objects. Not for arrays, since they already have a meaning

$ xidel -s -F "img[][email protected];type=local_pic;orig_img_a=orig_do_not_change;orig_q_jpeg=85;orig_q_png=6;create_preview=1;thumb_q=85;thumb_img=thumb_w;thumb_w=195;thumb_txt=thumb_txt_size" http://sendpic.org/upload.php -e '//fieldset[3]/input/@value'
This is supposed to happen?

Yes, ; is not used to separate parameters

And how would form() exactly work in this case? This at least doesn't:

You use {"file":..}:

 xidel  --verbose http://sendpic.org \
  -f 'form(
    //form,
    {
      "img[]": {"file": "download.gif"},
      "type":"local_pic",
      "orig_img_a":"orig_do_not_change",
      "orig_q_jpeg":85,
      "orig_q_png":6,
      "create_preview":1,
      "thumb_q":85,
      "thumb_img":"thumb_w",
      "thumb_w":195,
      "thumb_txt":"thumb_txt_size"
    }
  )' \
  -e '//fieldset[3]/input/@value'

Btw, a possible coloring bug?

Yes, it is highlighted as html when it should be xml

[Reino17]

I see. With --output-node-format=html the coloring is fine, but the indentation is not:

Also nice to see that only {"img[]":{"file":"download.gif"}} is really necessary. The other options are the defaults which form() already sends.

[benibela]

I see. With --output-node-format=html the coloring is fine, but the indentation is not:

The script element is the problem again. But that seems to be a bug in the XQuery standard. It says:

Whitespace MUST NOT be added or removed adjacent to an inline element. The inline elements are those included in the %inline category of any of the HTML 4.01 DTDs or those elements defined to be phrasing elements in HTML5, as well as the ins and del elements if they are used as inline elements (i.e., if they do not contain element children).

and HTML5 says:

Phrasing content is the text of the document, as well as elements that mark up that text at the intra-paragraph level: a, abbr, area, map, audio, b, bdi, bdo, br, button, canvas, cite, code, data, datalist, del, dfn, em, embed, i, iframe, img, input, ins, kbd, label, link (under body), map, mark, MathML math, meta, itemprop, meter, noscript, object, output, picture, progress, q, ruby, s, samp, script, select, slot, small, span, strong, sub, sup, svg, template, textarea, time, u, var, video, wbr, autonomous custom elements, text

[benibela]

Would it better to call such an option --post-raw, --raw-data, or --data-raw ?

[Baltazar500]

@benibela ,

Would it better to call such an option --post-raw, --raw-data, or --data-raw ?

It seems to me that the name "--post-raw" is more appropriate for the option. "--raw-data" or "--data-raw" is more curl-style :)

[Reino17]

I'm not sure I understand. You, @benibela, yourself said:

Xidel does not need all the request options of curl, because XPath can be used to build any request.

Hence, do we really need an extra command-line option like --post-raw? Is there a particular use-case that justifies this need? I thought Xidel could POST everything with -d or x:request({"post":"..."}).

[benibela]

Hence, do we really need an extra command-line option like --post-raw? Is there a particular use-case that justifies this need? I thought Xidel could POST everything with -d or x:request({"post":"..."}).

It might be hard to understand for people. And curl got a lot more data options recently, even a json data option

[Reino17]

I quoted you, but my question was actually targeted @Baltazar500. If he doesn't know such a particular use-case, then I think this issue can be closed.