If you check the Use custom form layout check-box, the familiar Layout editor will be displayed. The editor lets you define layout of the form that will be used for entering records into the custom table.

 

Clicking the Generate table layout button will generate a default layout that you can further modify.

 

Detailed information about the Custom tables module can be found in Developer's Guide -> Modules -> Custom tables.